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A project focused on further verification and 
development of a set of previously recommended vocational education 
indicators. These indicators were employer's satisfaction with the 
quality of the graduate's work, number of graduates employed in 
occupations related to program, labor market demand, number of 
graduates employed, training-related occupations, and program cost. 
One of the directions taken was further verification in terms of (1) 
current social and economic issues, (2) vocational education as part 
of education more broadly conceived, and (3) critical questions 
useful in interrogating any potential indicator. Another project 
direction focused on further development of indicators with adequate 
data available and proceeded through the steps of defining 
indicators, displaying information , sett ing benchmarks, and combining 
indicators. A third project direction involved analysis and 
recommendations concerning indicators without adequate data. Emphasis 
was on assessing the types of data available, directions of current 
research and development to improve the data, making best use of what 
is available, and suggest ing further ways of improvement . The two 
recommended potential indicators that served as content on which 
these developmental efforts were illustrated were labor market demand 
and train ing- re la ted occupations • ( YLB ) 
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PREFACE 



The authors express special thanks to the members of the Vocational 
Education Indicators Work Group who provide direction, constructive suggestions, 
critical assessment and motivation during the course of this research and 
development effort • The members were : Deena Allen, Robert Babcock, Ronald 
Dreyer, Craig Froke, Rosemary Fruehling, Helen Henrie, William Hohenhaus, Leon 
Linden, and Joe Malinski. Their attendance and involvement during work group 
meetings as well as in special assignments was most appreciated and "made a 
difference" in what can be reported. Hopefully the interaction of the Work 
Group and what is reported here will find their way into the larger process of 
developing a mission statement, key result areas, and indicators for vocational 
education currently on-going in Minnesota. 

This effort would not have^ been possible without the financial resources 
made available to the Minnesota Research and Development Center for Vocational 
Education by the Minnesota State Department of Education, Division of 
Vocational-Technical Education as authorized by the Minnesota State Board for 
Vocational Education. The encouragement of William Stock, our Project Officer, 
to involve practicing vocational educators in the effort was instrumental to 
whatever achievements are evident in this report and to a good deal of job 
satisfaction on the part of the authors. 

GHC' 
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ABSTRACT 



The work reported in this publication is a continuation of the efforts 
described in a previous report entitled, Potential Vocational Education 
Indicators; Vital Statistics for Planning, Review and Public Information 
About Vocational Education in Minnesota . Project efforts focused on further 
verification and development of a set of vocational education indicators 
which were recommended in the previous report. However, project results, 
while using the previously recommended indicators as "grist for the mill", 
should be useful in thinking about indicator development more generally. 

One of the directions taken during the project was further verification 
of indicators in terms of: (1) current social and economic issues, (2) 
vocational education as a part of education more broadly conceived, and (3) 
critical questions useful in interrogating any potential indicator. The 
conclusions based on this further verification were: 

• A mission statement for vocational education in Minnesota which is 
specific and address major value conflicts is essential to indicator 
verification. 

• A conceptual framework for vocational education which is consistent 
with the mission statement is necessary to provide a way of thinking 
and language (key concepts) with which to discuss indicator 
verification. 

• Indicator verification must address questions of the meaning of 
concepts used, the ends implied and means of operationalizing the 
indicators selected for use. 

• Indicators selected for general purposes in monitoring vocational 
education may need to be supplemented by indicators which are specific 
to particular purposes. 

• The process of indicator verification) must be continuous if indicators 
are to remain relevant to changing social conditions and technology. 

Another project direction focused on further development of indicators 
with adequate data available and proceeded through the steps of operational 
definition, information display, setting benchmarks and combining indicators. 
Results of the developmental efforts lead to the following conclusions: 

• Adequate operational definitions can be constructed for some 
indicators using existing information. 

v 
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• Displaying indicators involves considerations of both structure and 
format as a means to be understandable, focused and parsimonious. A 
structure and format- which seemed to meet these criteria was proposed 
and illustrated, 

• Setting benchmarks for judging the adequacy of vocational education 
involves choosing appropriate comparisons. Appropriateness depends on 
purpose, process used, desired degree of responsiveness to external 
factors, and judgments of what is reasonable. Benchmarks become an 
integral part of indicator display and a basis for subsequent 
interpretation • 

• Combining indicators allows focus on the inter-relationship between 
indicators and summary of general condition across several 
indicators. 

A third project direction involved analysis and recommendations 
concerning indicators for which adequate data was not available to move ahead 
with the steps of further development suggested above. From the analysis of 
two indicators concerning labor market demand and training-related 
occupations, the following conclusions were made: 

• While much labor market information is available, none is sufficiently 
specific, complete, or compatible with that available for vocational 
education programs for use" in indicator development. 

• It may be reasonable to use proxy measures of labor market demand 
derived from follow-up information on graduates of vocational 
education programs as a "stop-gap" mechanism in combination ^with 
existing labor market demand information until more adequate 
information is available. 

• Little progress can be made in using labor market information to 
develop a relevant vocational education indicator until a consistent 
and stable set of occupations related to each program is specified. 
The use of existing "crosswalks" between occupations and programs, the 
concept of "units of analysis" and existing follow-up information on 
occupations judged "related" by students offer good possibilities to 
resolve the relatedness issue at least in terms of the best means to 
use for this purpose. 

The overall results of pursuing these directions were the following 
recommendations for guiding further indicator refinement: 



The process of indicator verification and development must be 
continuous.. 

The need for both reflective thought and experience are important to 
indicator development. 

Some of the indicators must be pushed to the stage of actual use in 
planning , reviewing and providing public information. 
Involvement of those using and affected by use of the indicators is a 
necessary part of indicator development. 



vii 
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CHAPTER I. 
INTRODUCTION TO FURTHER 
INDICATOR VERIFICATION AND DEVELOPMENT 

The research and development activities described in this report are a 
continuation of those reported in the publication, Potential Vocational, 
Education Indicators; Vital Statistics for Planning, Review and Public 
Information About Vocational Education in Minnesota (Copa & Salem, 1982). 
That publication reported on the results of a two year effort to identify a 
first set of potential vocational education indicators for use in Minnesota. 
The process of identification included a comprehensive review of literature 
on the process of selecting and developing social indicators, a survey of 
vocational education administrators in Minnesota at the local and state level 
concerning their judgment of the importance of various indicators and a 
series of mini -conferences with groups of administrators to verify and 
discuss those indicators thought to be most important. In addition to 
recommending a set of "best" candidates for an initial set of vocational 
education indicators, authors of the report suggested that the selected 
potential indicators: (1) under go further justification and (2) be moved to 
a pilot testing phase to determine the feasibility of their calculation, 
display, interpretation and actual use. The purpose of the research and 
development efforts reported in this publication was to move forward in the 
further justification and pilot testing process for the recommended potential 
indicators. 

Process of Work 

Early in the planning of the further verification and development 
process for potential indicators, a decision was made to focus on the post- 
secondary vocational education level and to involve those who would 
eventually be responsible for deriving and using vocational education 
indicators. The focu3 on post-secondary vocational education was selected 
because of a high level of interest and pressing need (by legislative 
mandate) for the development of a set of indicators at that level to guide 
and formalize the planning process for vocational education programs, 
particularly, on a statewide basis. (Initial plans for this project had 
focused at the local level with a sample of schools and programs; these plans 
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were changed after discussion with and agreement by our Project Officer in 
the Minnesota Department of Education.) 

Involvement of vocational education managers was considered essential 
based on the review of efforts to develop social indicators in other fields 
of social endeavor. From this i-eview of past ideas and experience, -it was 
evident that if the indicators were ever to be accepted and carry some weight 
in actual decision making, those who would be using them needed to be 
involved in the verification and development process. Further, it was 
suggested that those who would be responsible for producing the indicators on 
a regular basis and who have a "working sense" of what is possible given 
available data and resources mu&t be involved if the importance, general 
purpose and nuances of the indicators are to be understood and worked into 
the indicator production process. Last, it was judged important to involve 
those who would bear the consequences of decisions made using selected 
indicators as a major decision making factor. These individuals would be 
particularly important to discussions concerning the adequancy of the 
indicators in covering important aspects of vocational education, in how the 
indicator is finally defined in terms of available data and in interpreting 
what the indicator communicates and hides. 

Using these guidelines, a Vocational Education Indicator Work Group was 
convened to assist in selecting a direction for the research and development 
efforts and to critically review the results of project work. The members of 
the Work Group were: 

• Deena Allen, Supervisor, Teacher Education, CETA & Vocational Equity, 
State Department of Education; 

• Robert Babcock, Specialist, Program Management and Budgeting, State 
Department of Education; 

• Ronald Dreyer, Management Information Specialist, State Department of 
Education; 

• Craig Froke, Assistant Director, Minneapolis Vocational Technical 
Institute; 

• Rosemary Fruehling , Manager , Post-Secondary Vocational Education 
Section, State Department of Education; 

• Helen Henrie, Technical Upgrading Specialist, State Department of 
Education; 

• William Hohenhaus, Agriculture Vocational Programs Specialist, State 
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Department of Education; 

• Leon Linden, Director, St. Paul Technical Vocational Institute; and 

• Joe Malinski, Planning, Research and Fiscal Control Director, Hennepin 
Technical Centers. 

The long and short term purposes of the vocational education indicators 
research and development effort conducted by the Minnesota Research and 
Development Center for Vocational Education were discussed by the Work Group 
at its first meeting. The short term purposes formed the context for Work 
Group discussions and project work during the three month period of the 
project. The purposes were as follows: 

Lonp Te rm: 

1 . Identify a set of vocational education indicators which should be 
used to plan, review und provide public information about vocational 
education in Minnesota. 

2. Develop and test procedures for producing and communicating the 
selected vocational education indicators. 

3. Provide assistance in using the vocational education indicators to 
maintain and improve vocational education in Minnesota. 

Short Term : 

1 . Select a series of "next-steps" in the identification and development 
of indicators. 

2. Begin to implement those next steps which are feasible and of high 
priority. 

The specific purposes of the Work Group in light of these overall 
project purposes was also discussed and agreed to at the first Work Group 
meeting. The purposes for the Work Group were as follows: 

1. Become aware of the progress which has been made on the development 
and use of indicators. 

2. Advise on the direction (next-steps) t.o be taken in further 
development and use of indicators. 

3. Become involved in at least cr.e aspect of the development or use of 
an indicator . 

The Work Group met three times during the course of the project, each meeting 
lasting approximatley two and one-half hours. The general process of work 
was for the Work Group to advise on project direction, for the research and 
development staff to make decisions on advice and follow through on project 
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activities, and finally for the Work Group to critically review the results of 
activities. The flow and focus of the process of work is reflected in the Work 
Group meeting agendas shown in Appendix I. 

Direction of Work 

A major first step in the Work Group's activities was to re:iew past 
research and development efforts related to developing vocational education 
indicators and to give advice on the specific direction of this project. The 
specific next steps for further vocational indicator development recommended in 
the previously cited report, Potential Vocational Education Indicators , were as 
follows: 

• Further verification of indicators', 

• Pilot testing of indicators with good data available, 

\ Developing data for indicators where data is not available, 

• Developing a history of use of indicators to assist in their 
interpretation , and 

• Developing a strategy to encourage use of indicators. 

In addition the report identified six potential vocational education indicators 
as meriting further work on the basis of their apparent importance to vocational 
education administrators. These indicators were as follows: 

• Employer's satisfaction with quality of graduate's work, 

• Number of graduates employed in occupations related to program, 

• Projected job openings in occupations related to program, 
a Number of graduates employed-, 

• Occcupations for which program is designed to provide training, and 

• Program cost. 

Using the above recommendations as a starting point, the Work Group 
deliberated on most feasible project direction given the time and resources 
available. The Work Group recommended going ahead in three directions: (1) 
further verification of indicators, (2) further development of indicators with 
adequate data, and (3) further' development of indicators without adequate data. 
These three directions provided the "advanced organizer" for project activities 
and, in seeming consistency, form the titles of the next 4 three chapters of this 
report. 

Within the direction of further verification of indicators, the Work 
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Group suggested and discussed several alternatives. The general purpose of 
this direction was to develop a greater sense of justification or confidence 
that the six recommended indicators were really important and whether or not 
other indicators should be added. Some of the alternatives discussed by the 
Work Group included: (1) relating indicators to* the future trends in social 
and economic development of Minnesota and the United States as opposed to the 
present or past, (2) testing the responsiveness of the indicators to present 
and immediate short term happenings in Minnesota (e.g., budget deficits, 
mobility of firms, unemployment, declining enrollments), (3) relating 
indicators for vocational education to education conceived more generally 
(e.g., What ought be the basic purpose of vocational education as a part of 
education? What alternative directions for vocational education are 
responsible choices?), (4) checking the responses of vocational education 
administrators concerning the importance of various indicators with those of 
other groups such as teachers, students, parents, employers, school board 
members, and legislators. 

From the comments and advice of the Work Group, it was decided to try 
two different strategies for further Indicator verification. Those were to 
examine the implications of: (1) national issues where vocational education 
was considered to have an important role, and (2) basic alternative 
conceptions or ideas of the purposes of education. A third strategy, 
included after the Work Group had concluded its meetings, and therefore not 
discussed by them, was a critical examination of the types of "questions 
needed to be raised about each potential indicator. The results of applying 
these strategies are reported in Chapter II. 

The direction taken within the area of further indicator development for 
those indicators with adequate data was most straight-forward. The next 
steps seemed to follow a logical order of implementation entailing: (1) 
operationally defining and deriving the indicator, (2) displaying the 
indicator, and (3) setting benchmarks for indicator interpretation. In 
addition, there was concern about the relationship among indicators and how 
to summarize their overall message; this resulted in an examination of means 
of combining indicators. It was decided that four of the recommended 
potential indicators had adequate quality of data for post -secondary 
vocational education program to merit further development as described above. 
These indicators were: (1) employer f s satisfaction with quality of 
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graduate's work, (2) number of graduates employed, (3) number of graduates 
employed in occupations related to program, and (4) program cost. The 
results of further development of these indicators are described in Chapter 
III. 

The last major direction concerned further development of indicators 
without adequate quality of data. Here the focus was on assessing what types 
of data were available, directions of current research and development to 
improve the data, making best use of what is available, and suggesting 
further ways of improvement. Two of the recommended potential indicators 
served as content on which these developmental efforts were illustrated. The 
indicators were: (1) projected job openings in occupations related to 
program and (2) occupations for which program is designed to provide 
training. Results of project work in this direction are reported in Chapter 
IV. 

What was learned about the continued development of vocational education 
indicators as one looks across all of the research and development work 
conducted as part of this project is described in Chapter V. Each of the 
recommendations made there should serve as motivation and direction for 
further programmatic work toward developing cogent vocational education 
indicators in Minnesota. 
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CHAPTER II. 
FURTHER VERIFICATION OF INDICATORS 



As described in the previous chapter, one of the directions taken in 
this research and development effort was to further question the 
appropriateness of the indicators selected as most important in the 
Vocational Education Indicators report. More specifically, it was decided to 
test the appropriateness of these indicators by three kinds of strategies. 
(There were several additional approaches which could have been used but time 
and financial resources did not permit their use.) The three strategies 
used , only two of which were reviewed with the Vocational Education 
Indicators VJork Group, were: (1) an analysis of current social and economic 
issues facing the nation and their implications for vocational education 
indicators, (2) an examination of various views of the purpose of education 
and their implications for indicators and (3) a critical analysis of a 
selected indicator — number of graduates employed in occupations related to 
a program (training related placement). While these verification strategies 
do not represent a comprehensive test of indicator appropriateness, they do 
provide at least a limited perspective of strengths as well as shortcomings 
to the indicator previously selected as important. 

Current Social and Economic Issues 

Vocational education in this country is currently undergoing a very 
detailed investigation and considerable thought in preparation for new 
federal authorizing legislation. In this context, those who practice 
vocational education have been developing a national agenda for vocational 
education over the next five years which relates vocational education to the 
important social and economic issues faced by our nation . The American 
Vocational Association (1981 ) has taken the leadership to bring vocational 
educators from all levels (e.g., teacher to administrator, local to federal) 
across the country, together with representatives of business and industry 
and other governmental agencies with an interest in employment and training 
to work on the agenda. The effort has taken 2-1/2 years, 10 ta3k forces, 
hundreds of individuals and many person -hours. This agenda, composed of six 
items, will be used to highlight how vocational education can be responsive 
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to major current social and economic issues. As such, it represents the 
direction likely to be taken by vocational education in the next few years. 
The items are not all new but are to receive renewed attention. For each 
agenda item, a brief "job description" of the problem to be addressed by 
vocational education will be provided. This is followed by a suggested set 
of indicators which might be used to monitor the degree of attention and 
impact of vocational education on the identified social and economic 
concerns. 

Strenthening Vocational Education in Depressed Communities 

The first agenda item for vocational education is strengthening 
vocational education in depressed communites. The job description is to 
improve the employment-related education of people living in economically 
depressed inner cities and rural communities — these are the places which have 
least capacity to improve on their own. Suggested indicators of attention 
and impact on this concern are: 

• Percentage of businesses rating vocational education important in 
relocation/expansion ; 

• Percentage of short-term programs requested by businesses for 
relocation/expansion ; and , 

• Change in funds for vocational education located in depressed areas. 

Meeting the Nation's Needs for Qualified Workers 

The second national agenda item for vocational education is meeting the 
nation's needs for qualified workers. The job description here is to prepare 
people in sufficient numbers and quality for the hard-to-fill jobs — 
shortage occupations. Some job titles which often fit this description are 
in the machine trades and computer technologies . Articles on the re- 
industrialization of America or bottlenecks to rebuilding our defense 
capabilities often cite the lack of trained people for these occupations. 
Other job titles likely to fit here are the fastest growing jobs (e.g., bank 
clerks, guards, industrial machine repairers, licensed practical nurses and 
teacher aides) and jobs with the most average annual openings (e.g., 
secretaries, retail sale workers, cashiers). Indicators for this agenda item 
could be: 
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• Percentage of students enrolled in programs leading to "shortage" 
occupations ; 

• Percentage of students in surplus occupations, and 

• Ratio of the number of graduates from a program compared with job 
openings in occupations related to the program. 

Keeping the Basic Programs of Vocational Education Relevant 

The third national agenda item for vocational education i3 keeping the 
basic programs of vocational education relevant . The challenge for the 
manager of vocational education here is to keep the staff, curriculum, 
equipment and facilities up-to-date with the changes in industrial 
technology. It is exactly the same challenge facing the manager of any 
enterprise concerning the issue of productivity . In keeping the basic 
program of vocational education most productive, some indicators might be: 

• Employer satisfaction with the quality of graduates' work; 

• Student satisfaction with the instructor, curriculum, and equipment; 
and 

• Percentage of program rated up-to-date by advisory committee members 
of evaluation teams. 

Conserving and Producing Energy . 

The fourth agenda item for vocational education involves its role in 
conserving and producing energy. The job description is to help develop the 
capacity to conserve energy and develop domestic renewable and non-renewable 
resources by translating the know-how evolving from research into widespread 
practice by industries and individuals. Two indicators for vocational 
education relevant for this agenda item might be: 

• Percentage of students enrolled in programs leading to employment in 
energy related industries; and 

• Percentage of students enrolled in programs leading to alternate 
energy /energy conservation occupations. 

Serving Groups with Special Needs 

The fifth agenda item for vocational education is helping groups that 
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need more (special) services to be successful in the labor market. These are 
the hard-to-employ (earlier the focus was on the hard-to-fill jobs); however, 
the focus here is on people. Some of the hard-to-employ include youth, 
displaced older workers, the handicapped, displaced homemakers, offenders and 
(resident) aliens. The indicator here might well be very specific to each of 
these groups: 

• Percentage of program enrollees with various special needs; 

• Dollars (or percent of budget) spent on special services; and 

• Number of special services. 

Responding to the Nation's Equity Goals 

The sixth and last item on the national agenda for vocational education 
is responding to the nation's equity goals . This job description means 
assisting in the process of obtaining better jobs for those who have been 
denied these jobs in the past because of race, sex or national origin. 
Possible indicators regarding equity in vocational education might be: 

t Percentage of enrollees with each equity concern; 

• Percentage of enrollees with equity concern in a representative sample 
of programs; 

• Percentage of enrollees with equity concern in programs traditionally 
not enrolling these students; and 

e Percent of programs in which enrpllees with equity concern are 
enrolled 25% more than their proportion of the student body. 

Discussion 

As one reads the indicators which have been posed for these agenda 
items, it is apparent that the information needs go well beyond, simple 
placement rates . There are information needs that are not likely to be 
answered by a simple computer printout of one data base. Rather there is a 
need for the integration of data from several sources, a narrative analysis 
by someone who is informed about vocational education and the larger social 
and economic culture of our nation, and in the end, it is likely to require 
face-to-face interaction between vocational educators and others involved in 
human resource development. While some of the indicators are the same as or 



related to the indicators previously selected as important, many are 
different and represent alternative positions as to what is important to know 
about vocational education. Differences in indicators might well be expected 
since the indicator's suggested in this section are tuned to particular 
purposes while those selected earlier were to be of a more general use. 
However, these differences should cause some concern and hesitance in 
"locking into" a set of indicators without careful thought as to what one has 
and has not. 

Vocational Education as a Part of Education 

It is assumed that vocational education is one part of a grander 
educational plan for individuals and society. If this is true, it is 
reasonable to examine the implications of one's educational philosophy for 
identifying important indicators for planning, reviewing and providing public 
information about vocational education. Further, examination of some basic 
alternative educational philosophies should provide a rationale for 
alternative visions or ideas of vocational education and corresponding sets 
of indicators. 

What follows is an analysis of the educational philosophies represented 
in the Vocational Education Indicators Work Group and a characterization of 
vocational education as it might be seen to best exist from the perspective 
of the different basic educational philosophies. A set of relevant 
. indicators for vocational education is posed for each philosophic 
perspective. 

Analysis of Educational Philosophies Represented in Indicators Work Group 

The Minnesota Analysis of Beliefs in Education (Minnesota Department of 

Education) questionnaire was developed to help people identify the basic 

nature of their educational philosophy (see Appendix II). This questionnaire 

serves as an appropriate means to sensitize one to the issues involved in 

characterizing "good" educational programs from the various educational 

philosophic perspectives (i.e., idealism, essentialism, pragmatism, 

existentialism, and behaviorism). 

The Work Group members were asked to complete the questionnaire; nine of 

' the eleven members responded. The results are shown in Figure 1. The fine 
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lines represent individual responses, while the bold face line indicates the 
average response for the group. Each of the philosophic perspectives is 
represented to some extent in the average for the Work Group and in each of 
its members; this is as would be expected. It would be rare for an 
individual to be purely of one philosophy or another. However, as is evident 
in Figure 1, the pragmatism, existentialism, and idealism perspectives are 
held to a larger extent than that of essentialism or behaviorism. A brief 
description of the basic tenents of each philosophy is provided in Exhibit 1. 

Vision of Vocational Education from Different Educational Philosophies and 
Implications for Vocational Education Indicators 

Each of the educational philosophies differ in their basic presupposition 
about the nature of "being," about "knowing," and about "values." These 
differences in presuppositions, in turn, lead to different responses to 
educational questions such as: (1) What is the naure of the learning 
process, (2) How does the student learn, (3) What is the most appropriate or 
relevant subject matter, (4) What is the best method of teaching, and (5) 
What ethics or aesthetics need to be learned? The responses to these 
questions from each of several philosophic perspectives were outlined by 
Lerwick (1979). He went further to characterize vocational education from 
each perspective in terms of clientele, relation to community, administrative 
staff, architecture or facilities, curriculum, teaching, method of teaching 
and participation and evaluation criteria. These characterizations of 
vocational education were posed as "useful fictions" to stimulate thought and 
discussion about alternative concepts of vocational education. 

The "useful fictions" as presented by Lerwick were drawn upon in this 
report in posing different visions for vocational education as a means to: 
(1) make specific to vocational education the consequences of holding a 
particular educational philosophy (as, for example, held by Work Group 
members) and (2) rationalize the very different possible reasoned visions for 
vocational education (and, therefore, the very different indicators important 
in planning, reviewing and providing public information about vocational 
education). This last • purpose is especially relevant to the process of 
verifying a particular set of vocational education indicators as is being 
suggested in this report. One is confronted with the question, "What 
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Exhibit 1 



Minnesota Analysis of Beliefs in Education 
Descriptions of Philosophical Positions 



The following are brief descriptions of the five philosphic positions 
used in the Analysis of Beliefs in Education inventory. While it is 
not possible to represent these five schools of philosophy adequately 
with a few statements, these general descriptions are provided as a 
means of underscoring the differences that exist among the 
philosophic approaches represented. There are many excellent texts 
available which detail these philosophical positions. 

Idealism 

This very early philosphic position maintains the dominance of the 
world of ideas over the world of sense reality. Therefor';, emphasis 
is given to mental discipline, the development of ideas and concepts, 
and the search for universal truths. 

It is the work of education, according to the Idealist to a3sist 
students in discovering the ideal truths that exist. This is 
commonly accomplished through such academic fields as history , 
mathematics, and literature. 

The Idealist believes that absolute and unchanging values exist. 
These values are timeloss and are found in each society and 
throughout history. Through the use of reason and logic, the human 
person can support belief in such values. 

Essentialism 

Essentialism takes its name from the fact that there exist certain 
essential knowledge and beliefs. This knowledge and these beliefs 
are knowable. 

Typically the Essentialist position unites the main beliefs of two 
other philosohpical schools of thought : Idealism and Realism. 
Realism is a philosophic position which advocates the objective 
reality of our world. 

Essentialism often has been linked with traditional religious 
beliefs. This results primarily from the Essentialist view of the 
student as a fugitive from learning — an assumption which correspond 
to the doctrine of original sin or the sin of Adam and Eve. In other 
words, human nature is thought to be inclined toward "evil." The 
instructor must practice consistent discipline in order to assist the 
student in developing the self discipline required for learning and a 
worthwhile life. 
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Exhibit 1 (Continued) 



Essentialist view the teacher as a master of learning. There is a 
knowable body of knowledge that teachers share with students in 
leading them to become well-educated adults. 

As with the Idealist position, Essentialism acknowledges the 
existence of absolute and unchanging values. 

Pragmatism 

This philosphic position is associated with the later progressive 
movement in American education. Pragmatists contend that truth is 
found in that which works. An idea is tested by examining its 
consequences; the degree to which the consequences are acceptable is 
the degree to which truth exists. 

The Pramatist does not simply advocate something that is seen to be 
useful for 'an individual. Rather, truth is that which helps us move 
toward what is progressively good for our community or society. 

In other words, truth . is relative and will vary according to time and 
place. Therefore, for the Pragmatist there are no absolute or 
changeless truths or values, other than perhaps Pragmatism itself. 

Reality, for the Pragmatist, is viewed as one's experiences. One's 
experience in trying to know or solve problems accounts for one's 
relationship is viewed as a process . In fact , the Pragmatist 
position is said to be process-oriented. 

In education, the Pragmatic point of view encourages students to 
learn through experiences of the real world , both physical and 
' social, and to solve problems associated with those experiences. 

Existentialism 

The Existential position focuses on the individual. Each person is 
thought to be unique and, therefore, efforts to impose group values 
upon the individual are not acceptable. 

In order to cultivate and maintain the unique character of the 
individual, a high degree of personal freedom is required. The 
person is not in search of some ideal or absolute truth. The 
individual strives to come to know his/her own being and person. For 
it is in knowing who the individual is that he/she can determine what 
reality is. 

Similarly , each individual, given personal freedom, determines the 
values that are important . It is obvious that this philosophic 
position is subject in its attention to the individual and to the 
individual's personal determina vaon of truth and values. 
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In education, the Existentialist assists students to create their own 
unique existence and being, and to recognize and practice personal 
freedom. Through personally selected experiences, the individual 
grows in self knowledge and achieves self- fulfillment. 

Behaviorism 

Two separable aspects of a system which is derived from psychology 
are: Behaviorism as a philosophic position and behavioral 
engineering as a technology. The philosophic position asserts that 
the human person is conditioned in all phases of life to respond and 
react in certain ways. Knowing this, it becomes clear that the human 
behavior is not free, but rather is determined by environmental 
stimuli. In addition to being controlled by such obvious stimuli as 
those provided in our physical environment, we are also controlled by 
the stimuli managed by other human beings. 

A stimulus can be either positive, as in a reward, or negative, as in 
punishment. It is believed that human behavior can be modified by 
establishing the "right" set of conditions. Certain members of 
society, usually those associated with the scientific/technological 
community, have the responsibility to identify the conditions which 
change behavior for the good of society and the individual. 

Behaviorists generally are silent on the topic of values. That 
behavior' which results in positively reinforcing stimuli is seen as 
good . 

In eduation, the Behaviorist is keenly interested in maintaining the 
right set of conditions for producing learning, e.g., a learning 
environment in which correct behavior is followed by positive 
consequences. Stating specific behavioral outcomes of learning in 
some quantifiable way and modifying student behavior toward learning 
reflect the usual concerns and activities of those who subscribe to 
Behaviorism. 



Source: Minnesota Department of Education. User's guide to 
accompany Minnesota Analysis of Beliefs in Education . St. Paul, MN. 



educational philosophy is represented in the indicators under verification? 
Is this the educational philosophy which was intended? And, what other 
educational philosphies (and their relevant indicators) should also be- 
included?" 
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Table 1 provides a brief characterization of vocational education from 
various educational philosophic perspectives and several indicators which 
seem consistent with each perspective. The labels given to the philosophies 
differs slightly between the Minnesota Analysis of Beliefs in Education 
questionnaire and the Lerwick report. The correspondence between labels is 
provided in Table 1. Lerwick added a sixth perspective, that of 
"reconstructionism, " which is not included in the questionnaire but is made a 
part of Table 1. 

Discussion 

The purpose of this strategy to verifying the appropriateness of 
indicators previously selected was to step back and view indicators from a 
larger and more basic context — that of education in total. The assumption 
was that one's vision of education has implications for the purpose held for 
vocational education, which in turn should consistently imply important 
vocational education indicators. This is a deductive approach to indicator 
verification. What is evident is that various educational philosophies 
suggest quite different visions for education (and vocational education and 
indicators). The analysis of Work Group participant's beliefs about 
education revealed that a combination of philosophies are held within and 
among even a small group of individuals concerned about vocational education. 
While acknowledging that an eclectic view seems to be held, it is alarming to 
note that the different educational philosohpies have conflicting basic 
presuppositions; that is, in the end when "push comes to shove" (as during a 
budget cutting process), the philosophies are not compatible and one would 
seem to have to go with one or the other. 

An important implication of using the above verification strategy is 
that a statement of the overall purpose of vocational education in Minnesota 
— a mission statement — on which their is consensus is critical to deciding 
if a set of vocational education indicators are appropriate. A mission 
statement must address the basic questions about beliefs in education (either 
directly or, if indirectly, with clear implications) such as: (1) What is 
the nature of the learning process, (2) How does the student learn, (3) What 
is the most appropriate or relevant subject matter, (H) What is the best 
method of teaching, and (5) What ethnics or aesthetics need to be learned? 



Table 1 

Characteristics of Vocational Education From Various Educational 
Philosophic Perspectives and Likely Corresponding 
Vocational Education Indicators 



Philosophic* 


Key features 


Note able 


Likely 


perspective 


(symbols) 


characteristics 


indicators 


Perennialism 


Excellence 


-Ideals should be taught 


-Employer 


(Idealism) 


(blue ribbon, 


-Craftpersonship 


satisfaction 




No. 1, A, 


-Quality, pride in work, 


with quality 




grin of con- 


durable products, quest 


of work 




fidence, high 


for perfection, beauty 


-Customer 




achievement, 


-Role of work in history, 


satisfaction 




good self 


culture, civilization 


with quality 




esteem) 


-Develop conscientious, 


of work 






moral, intelligent workers 


-Ratings in 






-Teachers with mastery of 


competitive 






skills, excellent role models 


technical 






-Students curious about 


demonstration 






larger issues 


events 






-Develop thinking about 


-Student satis- 






reasoning skills 


faction with 






-Evaluation based on stan- 


quality of work 






dards of excellense 


able to produce 


Essentialism 


Basics 


-Develop saleable skills 


-Percent employed 




(Paycheck, 


-Preserve stable work force 


-Percent employed 




oak tree, 


-Meet proven needs for jobs 


in program- 




basic tools 


available for military and 


related 




— hammer, 


economic security 


occupations 




typewriter , 


-Increase earning power for 


-Marginal increase 




cash register) 


those who can benefit 


in earnings as 






-Use advisory committee from 


result of 






business and industry to 


training 






identify genuine employment 


-Employer satis- 






opportunities 


faction with 






-Reality in the classroom 


quantity and 






(e.g., time cards, work 


quality of work 






orders, OJT) 


-Teacher /student 






-Concern with placement rates-, 


ratio 






cancel programs which fail 


-Program cost 






-Teacher technically up-to- 


-Occupational 






date 


demand for 



-Little innovation, stay with program graduates 
established occupations -Increase in tax 

-Evaluate on performance on revenue from 
the job higher wages and 

productivity. 



9 

ERLC 
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Tablo 1 (Continued) 
Characteristics of Vocational Education From Various Educational 
Philosophic Perspectives and Likely Cooresponding 
Vocational Education Indicators 



Philosophic 61 


Key features 


woteaDie 


Likely 


perspective 


v symbols; 


characteristics 


indicators 


. 

Prog ressiv ism 


Problem 


-ben -sufficient 


-Long term rate 


(Pragmatism) 


solver 


economically 


of employment 




'.Light bua.b, 


-relxiDie and adaptable to 


-Rate' of 




question mark , 


changing needs 


advancement 




lightning , 


-Maximize career options , 


-otudent satis- 




key) 


focus on transferable skills 


faction with 






• jC ai *ixng viewcn as in e 


being able to 






long process 


successfully 






— lnioi ra stuaents oi options 


deal with 






and consequences 


vocational 






-Experiment with real 


problems (e.g., 






problems, use laboratories 


finding suitable 






and community 


job, changing 






-Focus on program clusters 


work require- 






-Teacher as guide, knowledgable 


ments, occupa- 






of changing aspects of work 


tional mobility) 






-evaluate based on coping 








with vocational problems 




Existentialism 


Individual 


-Student accepts repsonsi- 


-Student satis- 




k contem- 


Dixity tor own aestiny, 


faction with 




plating 


can't learn for them 


program in terms 




person, 


-Develop student self- 


of own needs 




voting , me , 


awa reness 


-Student's feeling 




student, 


-Student as choosing agent, 


of self -manage- 




finger print) 


inner directed 


ment and control 






-Creative 


in dealing with 






-Student sets own goals, 


vocational 






personally responsible 


problems 






for choices 


-Number of 






-Knowledge is personal 


students self 






-Work is way of knowing self 


employed 






-Traditions, social demands 








not important 








-Student awake to themselves 








ana wnac s going on arouna 








them 








-Student preferences important, 








attentative to student needs 








(as defined by them) 








-Personalized instruction 








-Evaluation of program from 








students perspective 





Table 1 (Continued) 
Characteristics of Vocational Education From Various Educational 
Philosophic Perspectives and Likely Cooresponding 
Vocational Education Indicators 



Philosophic a 
perspective 



Key features 
(symbols) 



Noteab le 
charac teri'stics 



Behavioral 
Engineering 
(Behav iorism) 



Eng ineered 
(Flowchart , 
arrow, piece 
of puzzle, 
computer 
card) 



Likely 
ind icators 



-Desire planned, stable 

society 
-Students behaviors 

capable of modification, 

can be made to fit in 

(and enjoy it) 
-Measurable goals and 

objectives 
-Scientific testing and 

evaluation for selection 

and placement 
-Match individual and job 
-Rely on expert, pro- 
fessional to make choices 
-Goals derived from larger 

social plans and policy 
-Focus on "competency 

based" curriculum 
-Teacher responsible for 

materials, content, methods 
-Evaluate using pre-post, 

experimental-control designs 



-Percent employed 

in related 

occupations 
-Program cost 
-Bnployer satis- 
faction with * 

quality and 

quantity of work 
-Occupational 

demand for 

program graduates 
-Viability of 

alternative 

sources of 

training 



Rec obstruc- 


Reformer 


-Teach worker rights and 


-Student partici- 


tionism 


(Picket 


responsibilities 


pation in 


sign, 


-Help to see a better future 


work -re la ted 




■ broken 


-Lead change, consider what 


political and 




arrow , 


"should be" 


social issues 




sunrise , 


-Develop spirit of active 


-Student feelings 




globe, 


participation in social' 


of control over 




knight, 


change 


work life 




scale of 


-Develop leadership skills, 






justice) 


ability to think critically, 






to confront 
-Teacher as socially active, 






'I I 


involved in social changes 






related to work 
-Use real life situations 
in teaching 








-Student has role as change agent . 



a Philosophics labeled in parentheses are titles from Minnesota Analysis of 
Beliefs in Education. 
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The mission statement should answer the basic question: What is a 
vocationally educated person? From the minimal analysis conducted here, a 
mission statement which is abstract may be relatively easy to draft and gain 
consensus about; however, one which is more useful, especially at time's of 
crisis, because it is specific and has confronted the basic value differences 
in predominant educational philosophies held by vocational educators, will 
require time, serious dialogue and painful choice. 

Critical Analysis of a Selected Indicator 

That those leaving a vocational education program secure a job has 
consistently been accepted as an overall aim of vocational education. An 
additional refinement of this aim is that the job be one which is ''related" 
to the type of training program which the former student left. This 
additional refinement to the aim of securing a job implies that vocational 
education programs, at least at some point, are designed to provide training 
which is specialized to a job or group of jobs. It was not surprising that 
"number of graduates employed" and "number of graduates employed in 
occupations related to their training, program" were two of the indicators 
selected as most important to planning, reviewing , and providing public 
information about vocational education in Minnesota, 

When the intent of a vocational program is to provide specialized 
preparation for a job or group of jobs , then the aim of expecting program 
leavers to enter these training -re la ted jobs becomes appropriate. The 
justification being that of efficiency — presumably if resources are spent 
by both student and school in preparing the student for a specific job and 
then the student is either unable or does not want to secure a job of this 
kind (training -related job) , then the resources invested have been wasted to 
the exterit that the specialized training is not used. From an efficiency 
perspective in this situation, it would have been bet* u -er for the student to 
have chosen a different educational program (i.e., one related to the job 
eventually entered) or not to have entered an educational program at all, 



1 This section did not benefit from review by the Vocational Education 
Indicators Work Group . It was done by George Copa as a separate exercise 
related to a previous MRDC project to illustrate the issues which must be 
addressed in verifying the appropriateness of a specific indicator. 
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since the student was evidently able to secure the chosen job without 
additional, specialized preparation. 

Importance of Identifying Training -Related Jobs 

When the aim of vocational education is to provide more specialized 
training for a limited set of jobs or even a specific job as opposed to more 
general preparation of use in all jobs, then consideration of training- 
related jobs becomes important. First, in planning the vocational education 
program, it would then be important to identify the set of jobs for which the 
program is to provide an education. This delimitation of jobs would be 
needed in order to design an appropriate curriculum and secure relevant 
teachers and facilities. Another reason for preidentifying the specific set 
of jobs for which an education is to be provided is in order to assess if 
there is a market (labor demand) for persons with this type of an education. 
Prespecification of the training-related jobs acknowledges which specific 
jobs should be examined in terms of likely opportunities for employment. 

Second, it is important to identify training -re la ted jobs for the 
purposes of evaluation — when it becomes necessary to decide if the former 
student is in fact employed in a training -related job. This evaluative 
decision is appropriate for the efficiency ~ reason already presented. 
Presumably the same set of jobs considered to be training -related during the 
planning phase of a program is appropriate to the evaluation phase. 

Purpose of Verification Strategy 

The purpose of this verification strategy was to focus on the question, 
"How should one decide which jobs are related to a training program?" 
Somehow, this question must be answered directly and consistently before the 
aim of providing specialized training might be planned or evaluated with any 
sense of validity. The more specific questions to be addressed in focusing 
on the above general question are: 

• What conceptual framework could provide a useful structure for 
thinking about the concept of training -related jobs? 

• What issues must be resolved in identifying training -related jobs? 



-22- 



29 



Following through on this verification strategy is used to illustrate the 
categories of issues and process of thinking which should be used with each 
potential candidate for a vocational education indicator as a part of judging 
its appropriateness and feasibility. Because the indicator of training- 
related placement is commonly accepted as important to monitoring vocational 
education, the implications of this analysis are all the more salient, 

A Conceptual Framework For Thinking About Training -Related Jobs 

This section provides one way of thinking about the concept of a 
training -related job. As such, it identifies the major factors or variables 
important to the consideration of training -relatedness and the relationship 
among these factors. Much of the perspective provided here is drawn from an 
earlier report published as part of the Minnesota Research and Development 
Center for Vocational Education's program of studies dealing with improving 
the planning of vocational education (Copa, 1981). Without a conceptual 
framework describing the social phenomen of interest, there is little basis 
to systematically think about and serve as a basis for evaluating the 
appropriateness of indicators of the kind sought after here. 

Concept of vocational education . Vocational education is posited for the 
purposes of this analysis as a process. . ..as a means to an end rather than an 
end in itself. As a process, vocational education is viewed as having its 
major impact on individuals as opposed to jobs. Individuals serve as the 
major input to vocational education' and are somehow changed while involved 
with vocational education. The end toward which vocational education is a 
means is the enhancement of the mutual satisfaction of the individual and 
society, in general, through the individual's participation in a work role. 

Characteristics of individuals changed by vocational education . 
Individuals can increase their satisfaction in a work role through vocational 
education by exploring and identifying their needs , improving their ability , 
changing their geographic location or changing the time at which they will be 
available for a particular kind of work. Assessment of satisfaction by 
individuals can take place along these four dimensions before entering a work 
role (i.e., while in school) or while in a work role and contemplating 
whether to stay or move to a different work role. Needs refer to the 
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intrinsic and extrinsic motivational wants of an individual which can at 
least partially be filled by work (i.e., security, authority, creativity). 
Ability refers to the tasks which an individual can perform or has the 
aptitude to perform as it relates to the world of work (e.g., make a butt 
weld, discipline a child, take a soil sample). The same individual will move 
through different levels of ability related to his/her work role over time. 
That is, at the pre-school and elementary age, the individual may be just 
beginning the exploration of tasks relevant to various occupations. Later on 
in life, the individual will reach higher levels of ability which are 
prerequisite to a job. Still later, the individual may need to increase 
his/her ability in a given area, learn abilities in a new area, or simply 
refresh the abilities he/she once had. Location refers to where the 
individual is geographically located. Time refers to when the individual is 
available to participate in vocational education, and eventually, when he/she 
will be ready to work. These four characteristics of an individual are most 
relevant when considering the way in which vocational education can change 
the behavior of an individual relative to his/her work role toward the end of 
increasing overall mutual satisfaction of individual and society. A fifth, 
somewhat different characteristic of an individual, but nonetheless relevant 
to the process of vocational education, is the number of individuals with 
particular characteristics who are to be served by vocational education. 

Characteristics of jobs relevant to vocational education . In order for 
society to maintain itself and grow (improve) requires certain functions to 
be accomplished , One of these functions ' has to do with work in the 
production of goods and services needed by the .society. Society's work 
functions are made explicit in terms of labor market demand for jobs. The 
relevant dimensions of these jobs are number of positions, reinforcer 
patterns, requirements, location and time. Number of positions refers to the 
number of different work positions (jobs) of various kinds which society 
makes available in order to accomplish the production, processing and 
distribution of goods and services desired. Reinforcer patterns refer to the 
description of a job's potential at fulfilling individual needs (e.g., 
providing economic security, opportunity to exert authority, potential for 
using creativity). Requirements describe the tasks needed to satisfactorily 
perform a given job. Location describes where, geographically, the position 
is located and time refers to when it is available to be filled. These five 
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characteristics of a job are parallel to the characteristics of an individual 
most amenable to change by vocational education and thereby provide another 
part of the basis for planning and evaluating the effect of vocational 
education. 

Conceptual framework for training -relatedness . Combining the five 
characteristics described in the sections on characteristics of individuals 
changed by vocational education with the five characteristics of jobs 
relevant to vocational education results in the framework shown in Figure 2. 
This framework shows the parallel nature of the characteristics (e.g., needs 
versus need reinforcers, abilities versus requirements). Assessment of 
mutual satisfaction of individual and society with the individual's 
performance in a work role can be made by comparison of the five 
characteristics at a point in time. It is assumed that increasing 
compatability of characteristics leads toward increased mutual satisfaction. 

Vocational education enters this framework as a process located between 
the individual and job (work functions of society). Vocational education can 
also be thought of as mirroring the five relevant characteristics of 
individuals and jobs as shown in Figure 3. Using this latter framework, 
training-relatedness of a job can be envisioned as the relationship between 
the characteristics of the vocational education program and the 
characteristics of the job. In a sense, this is measure of the 
responsiveness of the vocational education program to the job (labor market 
demand) . 

~ Another notion stimulated by this framework is a similar relatedness 
concept which could be envisioned between the characteristics of individuals 
and the characteristics of the educational program — a training-relatedness 
to individuals as opposed to jobs. This concept o.? relatedness would be 
especially relevant to the student counseling process before entering and 
during attending an educational program. The concept describes the potential 
responsiveness of the educational program to the individuals to be served by 
the program. Interestingly, a common dilemma in vocational education now 
becomes readily apparent — whether to be more responsive to job 
characteristics or individual characteristics when they are in conflict. 
Correspondingly, when there is high relatedness of eductional program 
characteristics and job characteristics resulting in societal satisfaction 
(with employers often as proxy judge) with the program and, at the same time, 
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Figure 2. Relationship of characteristics of individuals 
changed by vocational education and character- 
istics of jobs relevant to vocational education. 



-26- 

33 



Individual 
Characteristics 






Educational 
Program 
Characteristics 




Job 

Characteristics 






Number 




-I 


r ) 






Number 
Available 




Available 






























Needs 










— *> 


Needs - 
Reinforcers 








— 


















Abilities 




[ J 






Requirements 


























Location 




v J 






Location 






— 




















Time 






a > 




Time 





























Figure 3. Relationship of vocational education program to charac- 

teristics of individuals and characteristics of jobs. 
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a high relatedness of educational program to individual characteristics 
resulting in student satisfaction with the program, the desired state of 
mutual satisfaction exists. 

There is one more concept of relatedness which can be derived from the 
framework shown in Figure 3. This is the relationship between the 
characteristics of individuals (before, during and after the educational 
program) and the characteristics of the job when the comparison of 
characteristics is made before the individual enters the educational program. 
The contrast in individual and job is usually called a "needs" assessment. 
The comparison during the educational program is termed a "progress" report 
or grading. The comparison after the individual has left the educational 
program is often termed an "evaluation" and usually is accomplished through a 
follow-up study of students and/or their employers. This latter "evaluation" 
is often used to assess the effectiveness and efficiency of vocational 
education — could the students secure and succeed in training -related jobs? 
Results of this evaluation can be used to validate previous judgements of 
what are training -related jobs — provided the students and labor market 
cooperate (i.e.> the students seek to enter these jobs and the labor market 
makes employment opportunities in these jobs available). 

Critical Issues In Identifying Training-Related Jobs 

This section v;ill 3erve to present a set of critical issues which need to 
be resolved in defining which jobs are related to a vocational ■ education 
program. As issues, they have moi'3 than one viable answer or response, each 
with some basis in reason. The issues cannot be ignored in developing and 
using a procedure to define training -related jobs — either they must be 
dealt with directly or are resolved defacto through the procedures selected 
and the use of resulting information. The issues are divided into three 
basic categories of questions in terms of their relevance to meanings of 
"training -related jobs", ends served by using this term and the means used to 
identify training -related jobs. The issues identified are not exhaustive of 
all issues relevant to the task of identifying training -related jobs but form 
an initial list which can be reformulated and added to with more study. An 
abstract graphical figure will be used to help communicate each of the issues 
as they are presented for discussion. 
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Issues concerning the meanings of training -related job . There are many 
aspects to the term "training-related job 11 • The purpose of this section is not 
to produce a single sentence definition of training -realted job (i.e., often 
termed an operational definition) but to describe a more comprehensive picture 
of just what is being implied when the term "training -related job" is being used 
in reference to vocational education. The issues concerning meaning are 
parallel for the vocational education program and the job, 

1 . Delimination of vocational education program . What is the starting and 
ending point of the vocational education program to be used in comparing a job 
in order to determine training -relatedness? Somehow the boundaries of the 
educational program must be prescribed. Is the educational program made up of 
one course, a series of courses, a series of independent study modules, and 
internship? Is the educational program to include only what happens in school? 
What about on-the-job training obtained before or while attending school or 
immediately after leaving school? Even within the education received in school, 
where does the vocational education program start and other education leave off? 
Just what makes up the vocational education program in question? Is it 
different for each individual? Each class? Each school? The focus of this 
issue is shown in Figure 4. 

2. Delimitation of . job . As with the issue of delimiting a vocational education 
program, there is a corresponding issue in defining a job. What are the 
starting and ending points for the activities to be identified as a "job" for 
the purposes of identifying training -related jobs. Are jobs made up of a series 
of tasks or duties? If so, what are the common tasks or duties that uniquely go 
together to make up a particular job? How is the line to be drawn between one 
job and another? The intent of this issue is depicted in Figure 5. 

3 • Dimensions of vocational education program description. How is the 
vocational education program to be described for purposes of determining which 
jobs are training -related? According to the ^conceptual framework provided 
earlier , these dimensions could include the extent to which the program 
addresses a number of jobs, various kinds which need to be filled, job 
requirements and job need reinforcers as well as the extent to which the program 
is sensitive to the time when the jobs will be available to be filled and where 
these jobs will be located. The potential dimensions of a vocational education 
program relevant to defining training -related jobs are shown in Figure 6. Are 
all or selected dimensions to be used?. 
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4. Dimensions of job description . And, how is the job to be described? 
Turning again to the conceptual framework presented earlier, jobs were 
described along five dimensions pointed out in the previous section on 
vocational education program description. These dimensions as shown in Figure 
7 are number of jobs available, job requirements, job need reinforcers, 
location of jobs and time of availability. Again, the issue is, "Are- all of 
these dimensions relevant when defining training -related jobs?" 

5. Stability of vocational education program description^ How should the 
common knowledge that training programs vary in each of the dimensions of its 
description from one year to the next, from school to school, from teacher to 
teacher (even within schools), and from class to class (even for the same 
teacher) be treated for purposes of defining training -related jobs? These 
variations go on all the time and may be substantial for programs with the same 
titles and general description. This issue is complicated by the emergence of 
individualized vocational programs where the student can "pick and choose" the 
aspects to be included in their own unique program — here the notion of 
standardized vocational education programs almost completely disintegrates. 
And, how should further discrepancies between what is portrayed in the 
vocational education program descriptions and what is actually taught and 
between what is taught and what is actually learned be dealt with in defining 
training -relatedness? This issue is depicted in Figure 8. 

6. Stability of job description. As with the vocational education program 
description there is also an issue of the stability of a job description. 
Again there is common knowledge that jobs with the same titles and even general 
descriptions vary from one type of firm to another, between firms producing the 
same goods and services from one geographic location or time period within the 
same firm, and within job positions in the same plant. There may also be a 
discrepency between the job description and what is actually done on the job. 
How are these unstable aspects of a job description to be reconciled for the 
purposes of defining training -related* jobs? This issue is presented 
graphically in Figure 9. 

7. Classification of vocational education programs. Given some description 
of vocational education programs (which is properly delimited and stable), the 
next issue in pursing meanings is that of classification. Classification 
becomes useful if there is to be any grouping or aggregation for communication 
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Job Need - 
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Figure 6. Dimensions of vocational education program description. 
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Figure 7. Dimensions of job description. 
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purposes (as in planning or evaluating or providing public information); if 
there is to be grouping, how should it be done for purposes of communicating 
training -relatedness of jobs? Which dimensions of the program should be used 
in the classification scheme? How should the categories be labeled? How 
many categories are necessary? The issue of classification of vocational 
education programs is shown in Figure 10. 

8. Classification of job s . Classification of jobs is also necessary for 
aggregation and subsequent communication purposes . How are jobs to be 
classified for facilitating the valid communication of the training- 
relatedness of jobs? Should jobs be categorized by occupation (similarity in 
tasks performed on the job) or industry (similarity in goods and services 
produced by firms in which jobs are found)? How should the level of job 
requirements be taken into account? What is an appropriate scheme for 
classifying jobs by need reinforcers provided, by geographic location, by 
time when jobs will be available? How are these multiple dimensions of a job 
description to be combined in a classification scheme? The issue of 
classification of jobs is shown in Figure 11. 

Issues concerning the ends served by using the concept of training- 
related jobs . Before automatically (without serious thought) accepting the 
notion of training -related jobs as being relevant to planning and evaluating 
vocational education program (i.e., the proportion of former students 
entering training -related jobs), it might be well to consider whose interests 
are being served by this concept and what values are either being supported 
or negated in the process. What follows are some of the issues which seem to 
be relevant to. thinking about the ends served by using the idea of training- 
related jobs in connection with vocational education. 

1 . Vocational education as specific or general. The very use of the concept 
of training -related jobs implies that some jobs should be more related than 
others to a particular vocational education program. In turn, this implies 
that it is more desirable, at least in some situations, for eductional 
programs to be specialized rather than providing general preparation which is 
transferrable to all or at least a large number of different jobs. What are 
the "situations" where training -relatedness is an important consideration to 
vocational education and when it is not. For example, a vocational education 
program with an aim of job orientation or exploration might well consider 
training -relatedness to a wide variety of jobs in the planning phases but not 



-34- 




Program Category 1 



Program Category 2 



Program 
A 



Program 
D 




Program 
G 



Program 
C 




Program 
Q 



Program 
X 



Figure 10. Classification of vocational education programs. 
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expect students to necessarily enter training -related jobs (because they 
found during exploration that they did not like or could not qualify for the 
job). The issue is "If specialization is started in an educational program, 
where is it to stop?" When is "enough" in the best interests of the student 
and the larger society? Is a separate education program needed for each job, 
however jobs are described? If not, to what extent should jobs be grouped or 
clustered toward achieving the ends desired of vocational education? An 
extension of this issue is to question the degree to which it is desirable 
for learning in vocational education to transfer to other life roles outside 
of the work role. The answer to these questions will necessitate the taking 
of a reasoned position on the ends to be served or the mission of vocational 
education. One position on this mission was presented earlier in this 
section and described in terms of increasing the mutual satisfaction of 
individual and society as it relates to the work function. The issue of 
specific versus general education for vocational education is graphically 
illustrated in Figure 12. 

2. Value of job . In the conceptual framework presented earlier, a job's 
description is assumed to be a direct description of a function that society, 
in general, considers acceptable and wishes to have done and is therefore 
willing to reward. Society is a convenient abstraction set up for the 
purpose of discussing the interests of all individuals taken collectively. 
Vocational education is a form of education conceived by these individuals. 
Might it not be part of the role of vocational education to raise questions 
about the desirability of some jobs as the basis for an educational program 
(from the interests of these same individuals taken collectively and the- 
values they share about what constitutes the "good life"). Some jobs or even 
aspects of many jobs may be degrading to individuals because of working 
conditions, the nature of tasks to be done, or the equity of the rewards for 
doing the job. Or at least,- it may not pay to invest time by student or 
school in preparing for some jobs through a formal educational program 
because there is no margin of benefit to specialized education in terms such 
as pay or opportunity to get the jobs. The issue relating to the value of 
. the job is shown in Figure 13. 
3. Job description as an accurate portrayal of societal needs . The issue 
here is whether or not the job description as it is finally communicated to 
the educational program and potential job holders is authentic in terms of 
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(A Cluster of Jobs) 

Figure 12. Vocational education versus specific or general education. 




Figure 13. Value of jobs versus justification for educational program. 
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societal needs. Or, have societal wishes somehow been thwarted by employers, 
thereby resulting in exploitation of employees. This issue is shown as 
Figure - 14. 

4. Program responsiveness to individual and .job . How responsive should the 
vocational education program be to the job? Does this responsiveness come at 
the expense of being responsive to potential students? Might not too much 
emphasis on identifying training -related jobs in the planning and evaluation 
of vocational education programs "tip the scale" in the favor of the job? 
But then perhaps the emphasis is necessary if the ultimate criteria for 
vocational education is that the student be able to get a training -related 
job. Who should be considered most adaptable in the labor market — 
individual (potential job seeker) , or employer (job description), or the 
vocational education program? Who should change? Does it always have to be 
the individual? Too great an emphasis on a static "matching mentality" (of 
individual, program and job) may allow vocational education to easily fall 
into the stance that the job description is fixed and that the educational 
program and individual are the aspects which most need to change (adapt to 
the environment). However, it may well be "that from an equity of opportunity 
or justice perspective, it is the job description or hiring practices which 
need to be adapted to better fit the individual. If the public schools are 
to play a major role in social reform as well as inculcation, then the job 
description cannot always serve as the boundaries or parameters for an 
educational program. The issue of balance in responsiveness for the 
vocational education program between individual and job is presented in 
Figure 15. ■ 

Issues concerning the procedure or means for identifying training- 
relatedness . Given consideration to the meaning held for training -related 
jobs and the desired ends for vocational education (and how these can be 
facilitated through using the concept of training -related job) then it is in 
order to think about the procedures or means for identifying training -related 
jobs. The issues presented in this section deal more specifically with the 
technical aspects of the problem of relating vocational education to jobs. 
1# Start with vocational education or job . In trying to identify jobs 
related to a given vocational education program, should the process be 
started by first measuring the educational program or job? Depending on 
where the process is initiated, the results may well be different. Starting 
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Vocational education program responsiveness to characteristics 
of individuals and job. 



with the educational program, one is likely to find several jobs related to 
1 the educational program which vary in degree of relatedness from very close 
to not very much. Also starting with the educational program will likely 
mean using a set of learning objectives as a basis or scale for measuring 
relatedness (i.e., Is knowing these learning objectives of benefit in 
performing the job?). Alternatively by starting with a job, one is likely to 
find several related vocational education programs, some of which relate very 
closely and some which provide only partial preparation for the job. Also, 
in starting from a job perspective, one is likely to start with a lot of 
tasks performed on the job or needs -reinforcers describing the job and to uoe 
these lists in examining vocational education programs. The list of learning 
objectives and the list of tasks may be quite different in their content and 
organization. The issue of starting with vocational education program or job 
is shown in Figure 16. 

2. How much is enough to call "related" . In examining the relationship be- 
tween an educational program and jobs, just when is there enough evidence to 
classify the program and job as being related? The degree of relatedness 
might be thought of on a scale (continuum) from zero (no relation) to one 
(perfect relation). Where along this scale should the line be drawn between 
what will be called related and unrelated jobs? Might the line be drawn in 
different places for different purposes? How precisely does relatedness have 
to be communicated — two categories, five categories, one hundred 
categories? The use of "how much is enough" to be called related is depicted 
in Figure 17. 

3. Common denominator for vocational educa tion program and job. The most 
difficult question in the process of identifying training -re la ted jobs is the 
working out of a way to translate the descriptions of vocational education 
programs and jobs. Should the language of the educational program or job 
description be used or is there yet another language which would be more 
useful as a common denominator of program and job? Even with an agreed upon 
language for the dimension of "abilities/requirements" such as using tasks, 
one faces the issues of level of training on the task (e.g., familiar with, 
can supervise others) and importance of task (e.g., frequency of use, 
proportion of time). The issue of a common denominator is shown in 
Figure 18. 

H. Collecting information on training -relatedness . How does one find out 
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Figure 16. Starting with vocational education program or with job. 
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Figure 17. How much relationship is enough to be training-related. 
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exactly what is taught in a vocational education program and exactly what are 
the characteristics of a job? Do you analyze course outlines and job 
descriptions, observe classes and job performance, or interview teachers, 
students, and administrators and employees and employers? Do you use a 
checklist, questionnaire, unstructured interview schedule? How do you 
validate and check reliability of your information? Besides how and from 
whom you collect information concerning relatedness, when should the 
information be collected? Before the vocational education program starts? 
Periodically thereafter? If for evaluation purposes you need information on 
the proportion of students entering training -related jobs, do you focus on 
the first job entered, first two jobs, jobs held in first two years, or 
entire career (perhaps depends on entry versus advancement emphasis to 
program)? In this instance, how do you deal with the situation where the 
student is qualified for training -related jobs but for some reason choses not 
to enter these jobs. The issue of collecting training-relatedness 
information is presented in Figure 19. 

Discussion 

As should now be apparent, a good dt...l of reflection might be in order 
before uttering the words "training -related job" in a research study or 
evaluation report pertaining to vocational education let alone using them to 
describe an important vocational education indicator. A similar process of 
identifying issues might well be done with several other concepts commonly 
used, seemingly unconsciously, in potential vocational education indicators 
(e.g., special needs students, equality of educational opportunity, job 
satisfaction, vocational). While resolution of these issues is an idealized 
end point, of more imi stance is that the issues be continuously raised and 
openly discussed by the parties affected. The reasoning is that any 
resolution is likely to be specific to a time and place and group of 
individuals —as such, it is at best temporary and partial; what remains 
important is to continue dialogue. Also apparent from the analysis of 

the concept of training -related jobs is the categories of issues which need 
to be addressed in verifying specific vocational education indicators. These 
categories concern the meaning of key words used in stating the indicator, 
the justification cf the values or ends 
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Figure 18. Common denominator for vocational education program and job. 
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implied by the indicator and selecting appropriate means to measure or 
operationalize the. indicator. In stating specific issues within each of 
these categories as a basis for discussion, the necessity of a conceptual, 
framework is highlighted. As was illustrated, this iframewcr-z: ssrre-; to 
provide a way of thinking about the issues and a language \ v^i--.:to 
communication to others. This is accomplished by the fnr ?ark. — ; ±ts 
identification of key sub-concepts (the objects and boudaries disou: ,ion, 
and their interrelationship; as such, it serves to give both context and 
focus to issue discussion. From applying the idea of categories of 

issues and a conceptual framework to the concept of training -related job, the 
shere complexity and seriousness in consequences of indicator verification is 
evident. It appears a process which ought not be avoided yet will never be 
completed. 

Summary 

This chapter has served to describe efforts at further verification of 
vocational education indicators as to their appropriateness in planning, 
reviewing and providing public information about vocational education in 
Minnesota. The strategies used were only a sub-sample of those initially 
suggested and discussed by the Work Group which provided direction to this 
project. However, the strategies which were carried through do represent 
quite different ways to probe the appropriateness of potential indicators. 

In looking across the three types of analyses conducted, the following 
summary points seem justified: 

• A mission statement for vocational education in Minnesota which is 
specific and address major value conflicts is essential to indicator 
verification. 

# A conceptual framework for vocational education which is consistent 
with the mission statement is necessary to provide a way of thinking 
and language (key concepts) with which to discuss indicator 
verification. 

« Indicator . verification must address questions of the meaning of 
concepts used, the ends implied and means of operationalizing the 
indicators selected for use. 



01 



• Indicators selected for general purposes in monitoring vocational 
education may need to be supplemented by indicators which are specific 
to particular purposes. 

• The process of indicator verification must be continuous if indicators 
are to remain relevant to changing social conditions and technology. 
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CHAPTER III. 
FURTHER DEVELOPMENT OF 
INDICATORS WITH ADEQUATE DATA 



Four of the proposed vocational education indicators appeared to have 
adequate data available to warrant continued development. This development 
involved deciding upon appropriate operational definitions of each indicator 
and displaying- the indicator in a clear and useful format. The four 
indicators for which these steps were implemented include: 

• Number of Graduates Employed; 

• Number of Graduates Employed in Occupations Related to Program; 

• Employer's Satisfaction with the Quality of the Graduate's Work; and, 

• Program Cost, 

In the following sections criteria for appropriate operational 
definitions will be suggested and operational definitions for each of the 
indicators will be proposed. Existing data from the Minnesota Vocational 
Follow-Up System and the Post Secondary Vocational Program Budget/Financial 
Report will be used for these operational definitions. For each indicator, 
an operational definition will be proposed, followed by an exhibit of. the 
source and presentation of its data as it occurs in either the follow-up or 
budget report. Following will be an analysis of the ability of each 
indicator's operational definition to adhere to the criteria suggested for an 
appropriate operational definition. Additional sections of this chapter deal 
with setting benchmarks for indicators and deriving composite measures of two 
or more indicators. 

Operational Definition of Indicators 

In order that the nature of operational definitions be described more 
specifically, a set of criteria delinating "good" operational definitions was 
developed. These criteria were used to guide the development of the 
operational definitions, to evaluate their completeness and worth, and to 
assure their accurate interpretation. « 
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Criteria for Operational Definitions 



Criteria for the operational definitions should reflect an overall 
purpose of facilitating clear, coherent communication. Operational 
definitions should be more specific and require less effort in explaining the 
meaning of an indicator than is provided by just the title or statement of 
the indicator itself. At the same time, operational definitions must 
describe the essence or important characteristics of indicators, assuming 
that the indicators have discoverable and describable characteristics. Aware 
of these purposes, the following criteria were suggested: 

• Be a measurable characteristic of the indicator (Ker linger, 1966; Ebel, 

1972); 

• Cover or encompass the essence of the indicator vBbel, 1972; Reynolds, 
1975); 

• Communicate a more specific and clear understanding (Kerlinger, 1966; 
Ghiselli, 196H; Van Dalen, 1979); 

• Describe the operations necessary to make the specified measurement 
(Kerlinger, 1966; Ghiselli, 1964; Sellitz, Jahoda , Deutsch , & Cook, 
1966); 

• Be manipulable arithmatically to enable summarizing the measure 
(Ghiselli, 196H); 

• Be feasible in terms of data availability, expertise required to 
derive , and cost ; and 

• Give consistent measures for different times, investigators and places. 
(Sellitz, et al., 1966; Van Dalen 1979). 

Using these criteria as a guide the following operational definitions for the 
four indicators were developed. 

Indicator: Number of Graduates Employed 

Proposed operational definition . Percent of graduates responding to the 
Student Follow-Up Questionnaire and indicating that they are presently 
employed by completing Section III, Employment Information - Present Status, 

of the Student Follow-Up. 

Source of data . The data are derived from the Job Information section 
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of the Follow-Up Questionnaire completed by the graduate . 
replication of that section in the questionnaire. 



Exhibit 2 is a 



Exhibit 2 

Employment Information Section of the Statewide 
Follow-Up Questionnaire 



III. EMPLOYMENT INFORMATION • Present Status 
A. Check only one of the following: 



| 1 J Employed j 3 | Unemployed - Looking for work 

[ Military | 4 [ Unemployed - Not looking for work because: (check one) 

| 1 | Disabled | 4 | Pregnant 

35 1 2 I ,nTrainin 9 | 5 | Change in Marital Status 

S Personal Enrichment, not | | 

interested in employment |JLJ Unw,,,,n 9 to move 



Report of da ta. The data is reported in Table 2 of the 1981 Follow-Up 
Report , As described in the report, Table 2: 

shows (1) how many of the graduates included in this report were 
employed one year after graduation, (2) how many of the graduates 
were employed at some time during the year but not at the time 
when the follow-up was conducted on^ year' after graduation, and 
(3) how many people were not employed at any time during the year ; . 
This group includes people who were u nav ai lab le f o r emp loy men t 
throughout the year as well as those who were unemployed . It is 
possible to determine the total number of people who were' employed 
at some time during the first ye^r after graduation. 

Indicator: Number of Graduates Employed in Occupations Related to Program 

Proposed operational definition / Percentage of graduates responding to 
the Follow-Up Questionnaire and classified in Category 1 or Category 2 of 
Table 3 of the Statewide Follow-Up Report . 

Source of data . The relatedness of the employment of graduates is 
determined by Educational Management Services (EMS), the company conducting 
the postsecondary follow-up study. EMS clerical staff members assign to each 
graduate's occupation, at the time of the follow-up, a Department of 
Education (DE) code number, considering the graduate ! s and employer's 
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Exhibit 3 is a copy of Table 2 of the 1981 Follow-Up Report . 



Exhibit 3 



Table 2 of the 1981 Statewide Follow-Up Report: 
Graduates Employment During First Year 



2. GRADUATES 1 EMPLOYMENT DURING 'FIRST YEAR 

CATEGORY NUMBER PERCENT 

, . : 1 " " V E MP CO YE d" AT TIME OF FOLLOW-UP ; ; : : ^ ; : 10879 :0: : 81.29 \./\: 
' 2. EMP . PRI0R"TO NOT AT FOLLOW-UP — - : ;; v ; 1609, >;,12.02 

-'V;. NEVER. EMPLOYE 0/Nb '..RESPONSE . . , ^,894 ; 6.-68 ; , . \ 

ToTaTs " ------ ioo ^ oo 



descriptions of the job held by the graduate. The graduate's description of 
his/her present employment is from Section IV, Part (2) of the Follow-Up 
Questionnaire and the employer's description of the graduate's job is a response 
to three questions at the end of the Employer Questionnaire. These sections are 
shown in Exhibit 4 and Exhibit 5. 

Exhibit 4 



Job Description Item on the Follow-Up Questionnaire 
of the Statewide Follow-Up System 



(2) Present Job 

(Job you are presently em- 
ployed in. if same as first job, 
writ* SAME) 



Firm Name , 



What kin'J of business or industry was this? 



InduitrlU Code 



Location 



rr 



59 61 



(For example: Retail supermarket, dairy farm, road construction) 

Firm Address . . — ■ 

City/State - 2, P — 



O.E. Code CTI *~I ..]_] ^/example: TV repairman, cosmetologist, electrician, farm operator) 

Occupational J j J 
Code ^ - 



I j □ 

i f» 7 r 

Wcr« you — (Mark one box) 

Employee of private company, business 
or individual, for wages, salary, or 
commissions 



Job Duties 1 ■ — -7 

(For exampleiType. file, sell cars, operate printing press, clean buildings) 

Immediate Supervisor ■ 



□ 



Federal government employee 

State government employee 

Local government employee (city, county, etc) 



Check one: 
78 

| 1 | Full-time job 
| 2| Part-time job 

Check one: 



| ) Job related to training t 
I I Job riot related to training 



Number of months in this job since 
graduation from vocational school. 

CD 80 [Ml 



Self-employed in own 
-~\ business, professional practice 
or farm 
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Exhibit 5 



Job Description Question on the Employer 
Questionnaire of the Statewide Follow-Up System 



Job Title 

( For example: TV repairman, cosmetologist, electrician, farm operator) 

Job Duties, 

(For example: Types, files, sells cars, operates printing press, cleans buildings) 

What kind of business or industry is this? 



(For example: Retail supermarket, dairy farm, road construction) 

; ; 

Report of data . The data for Exhibit 4 and Exhibit 5 are tabulated and 

presented in Table 3 of the 1981 Statewide Follow-Up Report (See Exhibit 6). 

As described in the report, Table 3: 

Indicates the employment status of graduates one year after 
graduation. The information is broken down into* seven 
categories: (1) the number of graduates employed in closely 
related occupations (they entered the exact occupations for which 
trained); (2) the number of graduates employed in broadly related 
occupations (they were employed in occupations somewhat related 
to the curriculum areas from which they graduated but not in the 
exact occupation for which trained); (3) the number of graduates 
employed in unrelated occupations (they were employed but in 
occupations unrelated to the curriculum areas from which they 
graduated); (4) the number of graduates employed, but for whom no 
present job information was available (no determination 
possible); (5) the number of graduates in the military ; (6) the 
number of graduates unavailable for employment (they had reasons, 
as documented in Table 4, for being unavailable for employment); 
and, (7) the number of graduates unemployed (they did not have 
jobs but were actively looking for jobs). 

Indicator: Employer Satisfaction with the Quality of Graduate's Work 



Proposed operational definition . Percentage of employers rating 
graduates employed in related occupations in the top half of their work 
group, indicated by checking response 1 or 2 of Section IV of the Employer 
Questionnaire of the Statewide Follow-Up System. 



-51- 



57 



Exhibit 6 



Table 3 of the 1981 Statewide Follow-Up Report: 
Employment Status One Year After Graduation 



. 3. EMPLOYMENT STATUS ONE YEAR AFTER GRADUATION 
. . EMPLOYED, UNEMPLOYED, OR UNAVAILABLE 



CATEGORY 


NUMBER 


PERCENT 


1. EMPLOYED CLOSELY RELATED 


7414 


55.85 


2. EMPLOYED BROADLY. RELATED 


1104 




3. EMPLOYED UNRELATED 


1834 


; 13.81: - 


4. EMPLOYED - NO PRESENT JOB INFO 


-Vv 488 • 


;•:•;>' • .3.67 • 


5. MIL ITARY ■ >. . ■■ , ■ . • \ 


v : 39. 


o;29 


6. UNAVAILABLE FOR EMPLOYED 


771 


5.80 


7. UNEMPLOYED 


1623 


12.22 


NO RESPONSE 


109 


** 


T 0 T A L S 


- ^ 13382 


• " . 100.00 • 



Source of data . The data is taken from responses to the questions of 
Section IV of the Employer Questionnaire as displayed in Exhibit 7 below. 



Exhibit 7 

Employer Satisfaction Questions on the 
Employer Questionnaire of the Statewide Follow-Up System 



IV. In comparison with other workers in the same work group, how would you rate the employee's over.all competency, effectiveness, 
proficiency, general over all work attitudes, and other elements of successful job performance? 



; J In [he Top V* 

3 In the bottom '/>' but not among the lowest V* 



I 2 s In the top ■/* but not among the top '/« 
[ 4*1 In the lowest V* 



Report of data . The data indicating the extent of employer satisfaction 
of graduates working in related occupations are found in Table 19 of the 1981 
Statewide Follow-Up Study Report, Table 19 shows: 
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the number of graduates whose employers rated them: (1) in the 
top 1/4, (2) in the top 1/2 but not the top 1/4, (3) in the 
bottom 1/2 but not the lowest 1/4, and (M) in the lowest 1/4 
categories compared with other workers in their work groups. 
Also indicated are the number who were unemployed, did not, 
respond, or were employed in unrelated jobs. The aspects judged 
were competency, effectiveness, proficiency, general overall work 
attitudes and other elements of successful job performance. 

These data are displayed in Exhibit 8 below. 

Exhibit 8 

Table 19 of the 1981 Statewide 
Follow-Up Report: Employer T s Overall Ratings 





OVERALL RATINGS 




CATEGORY 


NUMBER 


PERCENT 


1. IN TOP 1/4 


2549 


50.44 


2. IN TOP 1/2 - NOT TOP 1/4 


- > 1928 




3. IN BOTTOM 1/2 - NOT LOWEST 1/4 


^/:V-;; v : : 'V'd;: 447 v. 


• • ; ^•:V•8•84/^^•-^":••:^^v^ 


4. : IN LOWEST 1/4 


• ,. 129 ; : 


2.55 


UNRELATED JOB 


1834 




"NO JOB 


' 3030 " "" 


""''"'■**' ■ ' ■ • •••• L ""' "' 
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Indicator: Program Cost 



Proposed operational definition . Instructional cost per average daily 
membership (ADM). Instructional cost is composed of licensed and non- 
licensed teacher salaries and fringe benefits, profesisonal/administrative 
travel and purchased services. 

Source of data . The data for this indicator comes from the financial 
records kept by each school district and reported to the State Department of 
Education. 

R eport of data * The raw d&tta c *?e reported in the Post-Secondary 
Vocational Program Budget /Financial Report (PVPBFR) . Exhibit 9 is an example 
of this data. 
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Exhibit 9 



Example of Cost Data From The 
Post-Secondary Vocational Program Budget /Financial Report 



1 TPFMQFn 

L lotrot v 


NfiN-l TP ■- 


LICENSED 


N0N-LIC 


-TRAVEL. 


CAI AD TF C 

bALAKlu j 


CAI ADTCC 


FRG RFNF 
. nrvu dl.ui 


FRG RENF 
. rr\u pl.hi 


INST-ADM 


, /AUn . . . 


. / nun 


/ADM 

. / null . 


7 ADM '"' 


/ADM 


/ UD ■ 0 


■ .0 


30.4 




16.3 ' 


1 CI A R 
1314 • O 


.0 


89.6 


.0 


35.9 


1 Q7R 7 


. 0 


145.8 


.0 


53.8 


11QC c 
1 iyD ■ J 


.0 


48.6 


.0 


16.9 


01 Q ? 


62.4 • 


46.0 


10.2 


17.4 


IUJl • .7 


.0. 


98.8 


.0 


33.9 


1 197 4 

lie/ .4 


• 0 


10.8 


.0 


.0 . 


lfifi7 0 




30.4 


". . .0 


. . . .0 




.0 


21.6 


.0 


17.0 


pnn? fi 


.0 


44.6 


.0 


26.5 


179d 7 


229.1 


84.2 


37.8 


73.7 


ORG d 

yoo . h 


.0 


60.9 


.0 . 


16.6 


07/10 7 


fi 


117.9 


■"- - .0 


100.8 


IdUU.J , 


20.6 


48.0 


1.2 


4.9 


R77 9 
O / / . & 


do? fi 


61.8 


37.0 


15.2 


090 R 


PfiO fi 


49.0 


31.4 


4.3 


91 00 /l 


fi 


l?fi d 


.0 


64.8 


1 G7K 7 


fi 


8? R 

UC . O 


.0 


1.2 


1 Q7Q 0 


9Rd fi 

C04 ■ D 


Id 1 ! R 

A" J a O 


21.0 


4.0 


i /i o 

l4DD.y 


fi 


7fi 0 


.0 


29.6 


i/iii n 

1411. U 


d7 Q 
4 / • y 


70 fi 

' .7 « U 


8.2 


9.8 


1 70fi 0 

1/oU. o 


Vd7 d 

14 / • 4 


1fi9 fi 


.0 


21.3 


lbd4.b 


fi 


fiO Q 

- DO a J 




232.6 


101U. o 


fi 


dR 1 

*+0 a 1 


.0 


47.7 


1 0/1 7 0 
ld4/ . C 


fi 


7d fi 


!o 


2.9 


i noi A 

lUdl . 4 




55.2 


.0 


135:3 


1 G07 7 

loco, o 


m*? 7 

103 . / 


7"? R 


18.5 


35.8 


i /i 70 n 
i4oy . u 


fi 


105.9 


.0 


.0 


fi 


fi 


. 0 


.0 


.0 


. U 


fi 


0 


.0 


.0 


1 V77 1 
110/ . 1 


110 1 

1 1 U , 1 


81 Q 


6.6 


5.5 


i m 9 ft 


fi 

■ u 


77. 3 


.0 


8.5 


Q/l 7 fi 
04 J . U 


fi 

■ u 


43.4 


.0 


.0 


1097 .0 


. u 


fid 0 

04 ■ y 


fi 


34.6 


fi 


- fi 


.0 


.0 


.0 


• 1627.6 


j. .0 


119.9 


.0 


53.7 


1496!7 


1 .0 


88.2 


; .o 


124.8 


1043.1 


.0 


44.3 


■ - .0 


30.6 


721.3 


.0 


34.1 


.0 


43.2 


696.4 


.0 


31.2 


.0 


27.6 


1589.7 


.0 


58.2 


.0 


50.7 


924.8 


.0 


• 66.9 


•■ .0 


. 39.3 


2092.2 


2.7 


82.7 


1.4- 


18.0 


.1906/? 


.0 


. 112.8 


.0 


49.7 


1237^ 


.0 


63.7 


: .0 






.0 


.0 


.0 


4£ i 


i6,: :! 


.0 


100.6 


.0 


115.8 


.0 


50.6 


.0 


'28.7 



TRAVEL. 
PROF DEV 

. /adm 

7:9 

.0 
.0 

6.3 
• • 5.8 • • 
V 1.3 
2.4 
. 8.9 
7.7 
29.8 
13.3 
23.6 ■ 

;V- ; . .0. 

■:■ .o 

• .5 '; 

2.8* 

:o. 

.0 
.4 
3.4 

- 1.1 
■ .7 
.0 
6.1 
.0 
.0 
14.1 
14.2 
.0 
.0 
16.2 
8.9 
29.2 
.0 
.0 
3.1 
10.4 
.0 
4.5 
.0 
1.7 
- -18.7 
41.6 
.0 
10.8 
.0 
.0 
2.8 



OTHER PUR . OTHER - 
> SERVICES- EXPENSE 
/ADM /ADM . 



" K3 
9.8 
'28.8 
11.0 
109.7 ' 
56.9 
20.5 

. 67 .i ; 

23.0. 
126.5 
108.8 
40.9 , 
46.1 
• 8.9- 
8.1 ; 
22.8 
468.1 
4.8 
19.8 
, 28.3 V 
118.4 
104.9 : 
: .0 
5.2 
.0 
.0 
249.3 
256.8 
; .0 
. .0 
.0 
.0 
.0 
1.3 
( ^ .0 
.0 
r 7.6 
.5 
5.8 
44.8 
20.5 
- .0 
75.5 
9.8 
31.1 
.0 
9.2 
.0 



15.3 
34.8 
.0 
22.9 
22.3 
4.3 
18.3 
■j 7.9 
5.9 
52.7 
68.2 
258.3 
- 19.6 

■■vv. vo;. 

• .0 
15.4 
29.1 
4.7 
4.5 
17.1 
18.5 
17.0 
.0 
12.0 
2.7 
.0 
10.9 
26.0 
.0 
, .0 
9.5 
61.7 
26.6 
5.1 

: .o 

15.8 
.0 
• .0 
67.7 
14.5 
23.3 
63.7 
34.6 
11.9 
74.7 
.0 
12.6 
17.5 



O 

ERLC 



Analysis of Operational Definitions 



Seven criteria for operational definitions were identified in a preceding 
section. This section will be used to analyze the extent to which the 
proposed operational definitions fulfill the criteria (see Table 2). 

Table 2 

Ability of Operational Definitions of Indicators to 
Meet the Criteria ror Operational Definitions. 



Indicator 





Criterion 


Graduates 
employee; 


Related 
employ- 
ment 


Employer 
satis- 
faction 


Cost 


• 


Is a measurable characteristic 
of the indicator 


High 


High 


High 


High 


• 


Encompass essense of indicator 


High 


Medium 


Medium 


Medium 


• 


Communicate a more specific 
understanding 


Medium 


High 


High 


High 


• 


Describe operations necessary 
to make specified measurements 


High 


High 


High 


High 


• 


Manupulable arithmatically 


High 


High 


High ' 


High 


• 


Feasibility 

-availability 

-expertise 

-cost 


High 


High 


High 


Medium 


• 


Yield consistent measures: 
time, investigators, places 


High 


High 


High 


High 



Discussion , Summarized in . Table 2 are the results of evaluating the 

ability of the proposed operational definitions of the indicators 1 to meet the 

criteria for adequate operational definitions. Judgement of the ability of 

the operational definitions to satisfy the criteria are indicated as being 

either high, medium, or low. A judgment of high means that the operational 
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definition of a particular indicator meets a particular criterion relatively 
well. For example, the ability of the operational definition of the 
indicator, Number of Graduates Employed, is "high" as a measurable 
characteristic of the indicator, but only "medium" at being able to 
communitcate a more specific understanding because, although the definition 
is easily measured, it does not communicate the most specific measuring due 
to some inadequacy in the Follow-Up Report . The Follow-Up Report only 
reports on the graduates responding to a follow-up questionnaire; the lack of 
not including non-respondents increases the report f s lack of completeness and 
decreases its accuracy and both these facts detract from the specificity of 
the operational definition . As non-respondents increase , the operational 
definition will become less specific regarding the percent of graduates 
employed because there are more graduates whose employment status is unknown. 

Using these relative terms of high, medium and low most of the 
definitions were rated high in meeting the seven criteria. One exception, 
that of specificity, has already been discussed. Two other exceptions are 
discussed next. 

The definitions of the indicators, Related Employment and Employer 
Satisfaction, although not deficient, were not as strong as would be desired 
in embodying the essence of their respective indicators. The determintion of 
related employment, as presently done is based on the coding of employment by 
individuals whose expertise about occupations and vocational programs can 
vary widely. This can lead to incorrect designations, thus incorrectly 
analyzing the data and not communicating valid percentages of the relatedness 
of graduates 1 employment. 

The definition for employer satisfaction suffers from a problem with the 
nature of the question, that of asking the employer to rank the graduate. If 
the intent is to determine whether employers are satisfied with graduates 1 
work as a way of indicating effects of vocational education, then more 
information is necessary concerning the "other" employees with whom the 
graduate is being compared. An employer could rank a graduate in the upper 
half of an employee group composed of workers trained on-the-job or rank that 
same graduate in the lower, half of a peer group 'which also received 
vocational training but had had more experience, although doing the same kind 
of work, as sp6*?ifi v M on the Employer Questionnaire. For f MS ^CAPon, the 
data collected stouid indicate to whom the graduate is being tfompair^ <ftiher 
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than just other workers doing similar work. Additionally, there are 
contextual problems of the employers 1 criteria for ranking: personality^ 
differences; the nature of the work; and other significant factors which may 
enter into an employer's assessment of a graduate's work. For these reasons, 
the two definitions indicated above were only rated "medium" on their ability 
to meet the standards of this criterion. 

The operational definition for the indicator, Cost, was judged "medium" 
on the feasibility criterion because this data is harder to obtain, involves 
somewhat more expertise to determine and is a bit more expensive due to the 
difficulty in obtaining it. The cost data is not part of the 'follow-up study 
and it must b3 obtained from different financial reports and then calculated, 
while the data for the other operational definitions come from just two 
tables of one report. 

The cost definition was also "medium" at communicating the essence of 
the indicator. Instructional cost data mainly reflects instructor salaries 
and benefits, variables which are for the most part, based on tenure in the 
position. The data does not easily facilitate program or school comparisons, 
nor does it indicate a state's or school's commitment to a particular 
program. 

Displaying Indicators 

In preceding sections of this report, criteria for operational 
definitions have been suggested, operational definitions for four indicators 
have been "proposed , and subsequently analyzed for their ability to meet the 
criteria. In this section, formats for displaying the data using the 
operational definitions will be presented and analyzed for their ability to 
meet certain suggested criteria. 

Criteria for Adequate Display 

The overall intent for displaying the operational definitions' data is 
to communicate a "message" of the "state" or success of vocational education 
in Minnesota. To accomplish this purpose in an efficient manner is a high 
order objective of the display format. Burdening the reader with extraneous 
data or failing to provide sufficient information can lead to failure in one 
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of two ways. There could be a failure to communicate what is known, but more 
important, the reader could develop a false impression of vocational 
education. Aware of these concerns, the following criteria for display of 
indicators were proposed: 

• Understandable; 

• Focused; and 

• Parsimonious. 

If the display format is understandable , the essential message is 
communicated upon immediate observance of the display. Being focused means 
that the display, in addition to communicating the message, draws attention 
to the central aspect of that message. If the display is parsimonious, it 
will communicate the message, focusing upon its most salient feature, without 
"clouding" the display with unnecessary data. 

Display Structure 

The proposed display of the data for each indicator is likely to be 
multi-leveled in at least two dimensions. First each operational definition 
can be subdivided into finer and finer sub-components of the concept being 
measured (e.g., overall employer satisfaction, employer satisfaction with 
graduates 1 personal characteristics, employer satisfaction with graduates 1 
acceptance of responsibility). Second the degree of program aggregation can 
be sub-divided in several ways (i.e., across all programs in the state, 
across all programs of the same kind in the state, across all programs in a 
particular school). Given these potential dimensions of indicator data, the 
proposed structure for displaying operational definitions is as outlined in 
Table 3. 

As indicated in Table 3, the structure is one which initially displays 
data based upon the operational definition as proposed, and then upon more 
specific aspects of the operational definition. Using the operational 
definition of employer satisfaction as an example, Level I specificity would 
be the presentation of data indicating the percent of graduates ranked in the 
top half of their work group by their employer, and Level II specificity 
would be a display of employer ratings on two sub -components, personal 
characteristics and work characteristics. 
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Table 3 

Structure for Display of Indicators 



Level T. - Specificity matched to indicators operational definition 
(most aggregate) 

Stage 1 - Average for each operational definition of all programs; 
Stage 2 - Average for each program; and 

Stage 3 - Average for each school within particular program. 

Level II - Specificity matched to sub -components of indicators opera- 
tional definition 
Stage 1 - Average for each operational definition of all programs; 
Stage 2 - Average for each program; and 

Stage 3 - Average for each school within particular program. 

Level III - Specificity matched to sub-components of sub-components of 
operational definition 
Stage 1 - Average for each operational definition of all programs; 
Stage 2 - Average for each program; and 

Stage 3 - Average for each school within particular program. 



At Stage 1, the statewide average for the operational definition, or a 
sub-component of it, is given. In Stage 2, the different program averages., 
contributing to the respective statewide average is given. In. Stage 3, the 
school averages within the specific programs are presented. A diagram of 
this structure is presented in Figure 20. The stage hierarchy exists for 
each level of specif icty in definition. 

Display Format 

To further illustrate the presentation schedule, but more importantly, to 
illustrate the proposed format for displaying indicator information, actual 
data were gathered from the . 1980 and 1981 Follow-Up and Program 
Budget /Financial Reports and are presented for different levels and stages. 
The format was designed to meet the criteria for display discussed earlier. 

Level I, Stage 1 . Exhibit 10 is a presentation of the statewide 
averages for each indicator for the selected time period. 

Level I, Stage 2 . In Exhibit 11, averages of the programs for each 
indicator shown in Exhibit 10 are presented. In this case six programs, 
representative of various areas of vocational education, are used, although 
all the programs in the state could be listed. The data are presented so 



Stage 3 

School averages 
contributing to 

St age 2 program average 

Program averages of an indicator 
Stage 1 contributing to 

Statewide average statewide average 

of each indicator of an indicator 



Figure 20. Diagram of three stages making up structure of indicator 
display for each level of definition specificity. 

that each program average is seen in comparison to the statewide average for 
all programs. 

Level I, Stage 3 . In Exhibit 12 the same graphing format as used in 
Exhibit 11 is used to present the individual school averages for schools 
offering' a similar program. As with programs, although it is possible to 
present averages of all the schools offering a particular program, only six 
schools were used. 

Level II, Stage 1 . For each operational definition, it is useful to 
examine sub-components of the concept being addressed by the indicator. The 
operational definition of employer satisfaction is measured by the percentage 
of graduates ranked in the upper 50 percent of their work group by their 
employer. Sub-components of this measurement are the employers 1 rankings of 
the graduates on work characteristics and personal characteristics. The work 
characteristics: quality of work; quantity of work; job success knowledge; 
ability to operate equipment; and reading, verbal and computational skills — 
all are from the questions of Section II of the Employer Questionnaire. The 
results of these questions are then tabulated in Table 17 of the 1981 
Follow -Up Report . Exhibit 13 is a display of these Employer Questionnaire 
and Follow-Up Report items. 
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Exhibit 10 
Level 1 , Stage 1 Display 



Indicator: Number of graduates employed. 

Definition: Percentage of graduates employed at the time of the 
follow-up study. 
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Indicator: Number of graduates employed in occupation related to 
program. 

Definition: Percentage of graduates responding to questionnaire and 
employed in occupations broadly or closely related to 
training program. 
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Exhibit 10 (Continued) 



Indicator: Employer satisfaction with quality of graduate's work. 
Definition: Percentage of graduates ranked in upper half of employees 
performing similar work by employer* 
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Indicator: Program cost. 

Definition: Instructional cost per average daily membership , 
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Exhibit 11 



Indicator: Number of graduates employed. 

Definition: Percentage of graduates employed at the time of the 
follow-up study. 



Accounting 

Carpentry 

Child care 

Program Farm equipment 
mechanic 



Fashion 

merchandising 

Licensed practical 
nursing 



State average = 87% 
-d) 



(8) — ^ 
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average 



State average 
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~(2) (7) 
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— (3) 

% Above % Below 
average average 



1981 



(5) 

-(1) 

% Above 
average 



Indicator: Number of graduates employed in occupation related to program. 
Definition: Percentage of. graduates responding to questionnaire and 
**» ■' employed in occupations boradly or closely related to 

training program. 



Accounting 

Carpentry 

Child care 

Program Farm equipment 
mechanic 

Fashion (12). 
merchandising 



Licensed practical 
nursing 



State average = 66% 
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average 
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1980 



$ Above $ Below 
average average 



— (3) 



-(5) 



1981 



$ Above 
average 



•(12) 
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Exhibit 11 (Continued) 



Indicator: Employer satisfaction with quality of graduate's work. 
Definition: Percentage of graduates ranked in upper half of employees 
performing similar work by employer. 
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Indicator: Program cost. 

Definition: Instructional cost per average daily membership; 
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Exhibit 12 

Level I, Stage 3 Display for Carpentry Program 



Indicator: Number of graduates employed. 

Definition: Percentage of graduates employed at the time of the 
follow-up study. 



School 



Alexandria 
Austin 
Bemidji 
St. Cloud 



State average = 88% 
(9) 



(9>- 



(16). 



St. Paul (20)-- ^^1-. 

Suburban Hennepin 
North 



(8). 
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State average = 8k% 
„(8) 



1980 
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average average 



-(5) 

-(6) 

-(6) 



-(1) 
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average 



■(9) 
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Indicator: Number of graduates employed in occupation related to program. 
Definition: Percentage of graduates responding to questionnaire and 

employed in occupations broadly or closely related to 

training program. 



School 



Alexandria 
Austin 
Bemidji 
St. Cloud 
St. Paul 



State average = 66% State average = 66% 

" ~(2) 



(2)- 
(5) 

(4) 

(3) — 



Suburban Hennepen 
North 



% Below 
average 



(5)- 



.(5) (6). 



19M 
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average 
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average 



■(14) 



-(9) 



-(16) 



% Above 
average 



1981 
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Exhibit 12 (Continued) 

Indicator: Employer satisfaction with quality of graduated work. 
Definition: Percentage of graduates ranked in upper half of employees 
performing similar work by employer. 



State average = 8656 



Alexandria 
Austin 
Bemidji 
School St. Cloud 
St. Paul 



Suburban Hennepin 
North (15) 



% Below 
average 



State average = 8756 
(14) 



-(1) 



(5). 



■(14) 
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(12). 
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average 
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average 



•(13) 
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~(2) 
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average 



Indicator: Program cost. 

Definition: Instructional cost per average daily membership. 



Alexandria 
Austin 
Bemidji 
School St. Cloud 
St. Paul 



(128) 



Suburban Hennepin 
North 



State average = $1390 State average = $1428 

(231) -I (212) -| 

(112)— (238). 

-(49) 

(10). (234) 

(116) — (176) 

** ** ** 

$ Below $ Above $ Below $ Above 

average I average average average 



1980 



** 



1981 



*This/data for Bemidji' was missing from the State Department of Education, 
**Data'is given by district. Suburban Hennepin has two schools, and 
individual school totals were not given. 
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Exhibit 13 



Work Character 1 .sties Items of the Employer Questionnaire 
and 1981 Follow-Up Report 



Questions 



II. In comparison with other workers in the same work group, how would you rate the employee on each of the following 
characteristics? 



1. 


The quality of employee's work .*. ... 


above 
avefage 

□ 


.*vc' - ■»- 

E 


\ n (251 


2 


The quantity of employee's work 




m 


in (26i 


3 


The degree to which the employee possesses specific job related knowledge important to success 




m 


|T] 127) 


4. 


The degree to which the employee is able to operate the equipment and apparatus used on the job 




on 


Hi (2 «» 


5. 


The degree to which the employee possesses basic reading, verbal and computational skilis 




m 


[T| (29) 



Report 



17. EMPLOYERS' RATINGS OF SELECTED WORK CHARACTERISTICS 







ABOVE 


AVER. 


ABOUT AVER! 


BELOW AVER. 




ITEM 


NO. 


PCT. 


NO. 


PCT. 


NO. PCT. 


1. 


QUALITY OF 
WORK 


2689 


52.08 


2317 


44.87 


157 3.04 


2. 


QUANTIFY 
WORK 


2129 


41.21 


2748 


53.20 


288 5,57 


3. 


JOB SUCCESS 
KNOWLEDGE 


2091 


40.57 


2788 


54.09 


. 275 5.33 


4. 


CAN OPERATE 
EQUIPMENT 


2433 


47.27 


2600 


50.51 


144 2.21 


5. 


READ/VERBAL 
COMP. SKILLS . 


. 2102 


40.81 : 


2812 


-54.60 


236 4.58: 



The data for personal characteristics: acceptance of responsibility; 
punctuality; ability to work independently; willingness to learn; cooperation 
with co-workers; cooperation with management; compliance with rules; and 
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attendance at work ~ all are collected with the Section III questions from 
the Employer Questionnaire ?nd reported in Table 18 of the 1981 Follow-Up 
Report . Examples of each are presented in Exhibit 14. 



Exhibit 14 

Personal Characteristics K 3ms of the Employer Questionnaire 

and 1981 Jollow-Up Report 

Questions 

111 In comparison to others in the employee's work group, how would you rate the employee on each of the following characteristics? 

above about below 

a verag e a verag e a verag e 

1. Willingness to accept responsibility H [U (XI 1301 

2. Punctuality □ GO GO < 31 > 

3. Ability to work without supervision Q [XI S l32i 

4. Willingness to learn and improve Q S S 1331 

5. Cooperation with co workers Q [J] (XI (34) 

6. Cooperation with management m m S' 35 ' 

7. Compliance with company policies, rules, and practices m m s 1361 

8. Work attendance m a s«37» 



Report 



18. EMPLOYERS' RATINGS OF SELECTED PERSONAL CHARACTERISTICS 



ITEM 

1. ACCEPTS 
RSPNSBLTY 

2. PUNCTUALITY , 

3. WORK WITHOUT 
SUPERVISION 

4. WILLING TO 
LEARN 

5. CO-WORKER 
COOPERATION 

6. MANAGEMENT 
COOPERATION 

7. RULES 

. COMPLIANCE 

8. WORK 
ATTENDANCE 



ABOVE AVER. 

NO. PCT. 

2895 55.97 

3045 58.81 

2522 48.84 

3406 65.77 

3128 60.53 

3122 60.42 

2648 51.27 

3350 64*80 



ABOUT AVER. 

NO. PCT. 

2020 39.05 

1891 36.52 

2223 43.05 

: 1584 30.59 

1909 36.94 

1914 37.04 

2369 45.87 

1627 31.47 



BELOW 
NO., ; : 

257/ 

241 

418 

188 

130 

131 

147 

192 



AVER. 

I ;pct. 

: :4;96 
' 4.65 
8.09 
3.63 
2.51 
2.53 
2.84. 
3.71 
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A problem encountered using Table 17 and Table 18 for sub-components of 
data obtained from Table 19 of the 1981 Follow-Up Report was that the data 
are categorized differently, precluding comparison between overall employer 
satisfaction and each of the sub-eomponents. When determining employer 
satisfaction from the data in Table 19, satisfaction was the percentage of 
graduates rated in the upper 50 percent of their work group by their 
employer. In Table 17 and Table 18, the classifications: above average; 
about average; and below average — all are used. It is difficult to compare 
these data to those of Table 19, although the ratings on the work 
characteristics and personal characteristics are sub -components of overall 
employer satisfaction. It is also necessary to make an arbitrary decision 
regarding how the Table 17 and Table 18 data explain employer satisfaction. 
Should satisfaction be above average or should it also include those 
receiving average ratings? Whatever the decision, neither choice will result 
in comparable data to that of overall employer satisfaction; therefore, it is 
only possible to consider data for the various stages of display somewhat 
independently. Further, employer satisfaction as displayed here will include 
all those rated above average and about average. This is based upon the 
assumption that an employer can be satisfied with an average worker. Using 
this method of calculation, it is possible to calculate employer satisfaction 
for the two sub-components of overall satisfaction. The results of these 
calculations are presented in Exhibit 15. 

Analysis of Display Structure and Format" 

In a previous section three criteria were suggested for analyzing the 
display of the indicators. Table 4 summarizes an analysis of the proposed 
display structure and format using those criteria. 

As observed from Table 4, the display format adhered to the criteria. 
The greatest advantage of the display was the overall conceptual structure 
involving the different levels arid stages within levels. In this manner, one 
could analyze a program in all the schools in which it is offered (Stage 3), 
or all programs, for a particular indicator (Stage 2), by examining a 
particular graph. Should it be necessary to examine the statewide average of 
an indicator from year-to-year that is also available without distracting 
data (Stage 1). 
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Exhibit 15 
Level II, Stage 1 Display 



Indicator: Employer satisfaction: work characteristics of graduates. 
Definition: Percentage of graduates about average or above average in 
work characteristics. 



100 



Percentage 90 
of graduates 




1980 



1981 



Year 



Indicator: Employer satisfaction: personal characteristics of graduates 
Definition: Percent of graduates about average or above average in 
personal characteristics. 



100 



Percentage 90 
of graduates 



1980 1981 
Year 



The Stage 2 and Stage 3 formats stress a comparison of programs or 
schools with an average rather than change from year-to-year, although it is 
possible to follow, horizontally, the pattern of a particular program or 
school. This emphasis reflects the assumption that it is more importa /c to 
know how a program differs from all other programs each year or how a program 
differs in different schools than over a period of years. The power of this 
format was echoed in the Work Group discussions of these displays in the way 
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Table 4 

Ability of Display Format Structure to Meet Suggested Criteria 



Display structure 





Level I, 


Level 


II, 


Level 


I, 


Level II, 


Overall 


Criterion 


stage 1 


stage 


2 


stage 


3 


stage 1 


scheme 


Understandable 


Yes 


Yes 




Yes 




Yes 


Yes 


Focused 


Yes 


Yes 




Yes 




Yes 


Yes 


Parsimonious 


Yes 


Yes 




Yes 




Yes 


Yes 



it drew their attention to important characteristics of the data, seem clear 
with minimum explanation, and followed a logical path of their thinking and 
questioning about what was being communicated. 

Although not a part of the analysis for the display scheme, the example 
of Level II, Stage 1, illustrates the inadequacy of the existing data base. 
Since different response patterns are used for the questions generating the 
data for overall employer satisfaction than for particular sub-component 
characteristics of graduates, there can be no comparison between the overall 
indicator and its sub-components. A standardized response pattern could 
solve this problem, and increase the usefulness of the data. 

Benchmarks for Indicators 

Implicitly, the topic of benchmarks for vocational education indicators 
has already been partially addressed in previous sections of this chapter, 
particularily the section on indicator display. Ia this section, the 
discussion will be more explicit as to benchmark definition, considerations 
in setting benchmarks, and ways to go about doing it. 

Definition and Purpose of Benchmarks 

In Webster's New Collegate Dictionary (1979), a benchmark is defined as: 
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\. k point of reference from which measurements may be made; and 

2. Something that serves as a standard bv which others may be measured, 

B^th rt^finitions imply the meaning that a benchmark is a standard or 
reference point for diking a measurement, which in turn presumably will be 
used to make a judgement of some consequence. That judgement will be based 
on an interpretation of the differences between the benchmark and the actual 
characteristics of the object, organism, or phenomenon under observation. In 
the case being described here, the phenomenon under observation is vocational 
education in Minnesota. The characteristics of concern are those described 
by selected vocational education indicators. The actual characteristics are 
as described through the operational definition of indicators and display of 
data pertaining to a particular time and place using these definitions. 

In order to make judgements about the "health" of vocational education, 
it is necessary to know whether or not the status of its important actual 
characteristics is adequate or acceptable, A common approach to judgements 
of this kind is to make a comparison; the question is, "Comparison to what?" 
And after comparison, "How large a difference is important to be attentive 
to?" The first question requires setting benchmarks of. the kind described 
above. The second question can be partially resolved by where the benchmark 
is set. That is, the benchmark can be set at a point above which (as for 
program cost) or below which (as for employment rate) any discrepancies are 
important enough to merit attention. 

Factors to Consider in Setting Benchmarks 

There are many factors which enter into the process of setting 
benchmarks. Some suggested by the Work Group are as follows: 

• Purpose for which benchmarks will be used. For example, should bench- 
marks be set differently for purposes of different groups such as the 
state department of education versus local boards of education or for 
different functions such as creating a political image versus funding 
decisions versus selecting programs to be revitalized? 

• Process used to set benchmarks. For example, who is involved in the 
deliberations, what is the appeal process for those not meeting the 
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benchmark, what are the attenvating circumstances which will be 
considered, are the benchmarks set in policy, how easy is it to change 
the benchmarks, what is the process for change? 

• Responsiveness of benchmarks to changing economic and social 
conditions. For example, how are the benchmarks responsive to 
economic downturns or upturns , differences in geographic areas , 
differences between programs in .serving the hard to train and employ, 
financial resources available to operate programs, - declining 
enrollments, changes in ages of idents, and changes in industrial 
technology? Many of these factory are cut of control of vocational 
education teachers and managers. 

• Relation of benchmarks for vocational education to those of others 
involved in similar activities. For example, how are the benchmarks 
related to those used for training conducted by private vocational 
schools, on-the-job colleges and universities? 

• Reasonableness of benchmark. For example, is it reasonable to expect 
that 100 percent of graduates should be employed in an occupation 
related to their training? Is it reasonable to expect 100 percent of 
employers to be above average in their satisfaction with graduates 
work? It is not even possible for all programs to be below average 
in program cost! If these extremes are not possible, what is 
reasonable? 

The above factors would suggest that the process of setting benchmarks 
will not be easy. Rather, what is needed is some experience with different 
types of benchmarks used for different purposes over a period of time. 
During this experience, serious reflection and dialogue should be focused on 
what seems to "work 11 in view of the many and complex considerations involved. 
As with indicators, the setting of benchmarks must involve delimiting a 
process of doing the setting as much as the actual settings. Both must 
remain continually open to questioning, discussion and change to insure 
improvement . 

Ways of Setting Benchmarks 

There are several approaches to setting benchmarks some of which are 
already implicit in the indicator displays presented in the previous section. 
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For example benchmarks can be established by comparison of a program to the 
average (or other measures of central tendency) of other programs of the same 
kind (see Exhibit 12). Or, comparisons can be made to the status of the same 
program in previous time periods (see Exhibit 12 again) as a way to detect 
trends or, more importantly, changes in trends. Or comparisons can be made 
to different types of programs for the same time period (see Exhibit 11). 
These benchmarks could be made more sophisticated by using ranges rather than 
fixed points or indexing the benchmarks to economic conditions such as the 
growth in Gross National Product or the unemployment rate* Each of these 
approaches may be appropriate for different purposes of comparisons. Some of 
these considerations have been attempted to be sorted out in developing the 
display structure and format already proposed. 

Yet another approach would be to set more absolute types of benchmarks 
which are largely based on considerations external to the nature of the 
programs. For example, it might be decided that at least one half of 
graduates ought to be employed if the program is to be considered at all 
viable on the basis of "just" general principles. Another approach to more 
absolute benchmarks used by economists in a cost-benefit sense is to focus on 
rate of return to investment in a particular enterprise. In the case of 
vocational education, the benchmark discussion would center on the return on 
investment in training to students and the larger society. Does a dollar 
invested in a particul^.i- vocational education program return the same amount 
as in another program? Is the return at least as much or more than a dollar 
invested in other income producing investments such as health, general 
education, transportation or economic development? Here it is possible to 
use the same benchmark across a wide variety of investment options. 

Combining Indicators 

Sometimes for the sake of brevity and focus it is most useful to combine 
indicators in one chart or even in one composite indicator (analogous to the 
consumer price index). Each of these strategies, contrasting tv?o indicators 
and developing a composite index, requires a different approach. 
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Contrasting Two Indicators 



In examining the relationship between two indicators, the approach is to 
chart them on the same diagram using the criteria for display discussed 
previously. For example, suppose one wished to assess various program 
offerings in terms of the indicators of placement in related occupations and 
employer satisfaction with the quality of graduate's work. The information 
concerning these two indicators for the vocational education programs 
examined earlier in this section is shown in Table 5. The information could 
be plotted as shown in Figure 21 in order to see the relationship between the 
indicators. Using the statewide average as a dividing line, four groups of 
programs emerge: 

• High related placement/high satisfaction (Accounting, Farm equipment 
mechanics) 

0 High related placement/low satisfaction (Nursing, Carpentry) 

• Low related placement/high satisfaction (Child care) 

• Low related placement/low satisfaction (Fashion merchandising) 
Using this inforamtion, it may be possible to better prioritize programs in 
terms of effectiveness. Other lines could be added to reflect acceptable 
ranges or benchmarks for indicators (minimum and maximum) . Similar plots 
could be constructed for other combinations of indicators. For example, 
program cost might be used as the horizontal axis for joint consideration 
with a number of other indicators to focus on cost-effectiveness of programs. 

Developing Composite Indicators 

At times, for the sake of simplicity and focus, it is most useful to use 
one overall indicator in discussions concerning the reviewing, planning, and 
providing of public information about vocational education. When this is the 
c^sc, it may be most appropriate to use a combination of indicators rather 
tnan select one indicator alone . The process of developing composite 
indicators will be illustrated below for two situations, one which uses very 
imprecise information about an indicator and the other very precise 
information. 

Operating with imprecise information . Using the information from Table 
5 for the indicators of related placement and employer satisfaction with the 
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quality of graduate's work and with the statewide average used as a cutting 
line between high and low, the following rating is possible to develop: 

High = above average = 1 

Low = below average = 2 

Table 5 

Related Placement and Employer Satisfaction with Quality of Graduate's Work 
Indicators for Selected Sample of Post-Seondary Vocational Education Programs 

Statewide Average, 1981 



Program 


Related 


Employer 




placement 


satisfaction 




indicator 


indicator 


Accounting 


66% 


90% 


Carpentry 


66 


86 


Child care 


37 


92 


Farm equipment 






mechanic 


68 


95 


Fashion merchandising 


51 


83 


Nursing (LPN) 


75 


82 


Statewide average 


63 


87 



100 



80 



60 

Related 
placement 
(*) 40 



20 „ 



Statewide 
average 



Nursing 
Carpentry 




Fashion 
merchandising 




20 



40 



60 



80 




Accounting 



Farm equipment 
mechanics 
Statewide 
average 



-Child 
care 



100 



Employer satisfaction with quality 
of graduate's work (50 

Figure 21. Relation of percent related placement indicator 

and employer satisfaction with quality of graduate's 
work indicator. 
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Then, the needed information on the sample of selected programs can be 
derived as shown in Table 6. 



Table 6 

Imprecise Ratings of Related Placement and Employment Satisfaction 
with Quality of Graduated Work Indicators 





Related 


Employer 


Program 


placement — 


satisfaction 




rating 


rating 


Accounting 


1 


. 1 


Carpentry 


1 


2 


Child care 


2 


1 


Farm equipment 






mechanic 


1 


1 


Fashion merchandising 


2 . 


2 


Nursing (LPN) 


1 


2 



This information showing an imprecise rating could be developed into a 
composite indicator by simply adding the ratings for the two indicators for 
each program. The results of this process are shown in Table 7. Using this 
new, composite indicator, programs could have a rating of values 2 to 4, with 
4 being lowest. With the composite indicator , each individual component 
indicator is being weighted equally. If this were not desired, and for 
example, related placement was considered much more important than employer 
satisfaction, the related placement indicator might be given a weight of two. 
Using the weighting procedure gives the results shown in Table 8. Notice 
that the composite indicator now ranges from 3 to 6. 

Operating with precise information . If detailed information about each 
of the indicators is readily available as is evident in Table 5, the 
indicator information can be converted to index numbers and then combined 
into a composite index . For example, related placement could be indexed such 
that 100 is set as the statewide average for all programs (i.e., 63% in this 
case). Given that a related placement of 63% is indexed at 100, then a 
related placement for accounting (66%) is equal to 100(66)/63 = 105. The 
same process could be used with the employer satisfaction with quality of 
graduated work indicator. If this process were carried through with the 
information in Table 5, it would appear as in Table 9. 



Table 7 

Composite Indicator of Imprecise Rating of 
Related Placement and Employer Satisfaction with 
Quality of Graduated Work Indicators 



Program 



Related 

placement 

indicator 



Employer 
satisfaction 
indicator 



Composite 
indicator 



Accounting 
Carpentry 
Child care 
Farm equipment 

mechanic 
Fashion merchandising 
Nursing (LPN) 



1 
1 

2 

1 

2 

1 



1 

2 
1 

1 

2 
2 



2 
3 
3 

2 
4 

3 



Table 8 

Weighted Composite Indicator or Imprecise Rating of 
Related Placement and Employer Satisfaction with Quality 
of Graduated Work Indicators 





Related plac 


ement 


Employ 


3r satis 


faction 


Composite 




Rating 


Weight 


Weighted 


Rating 


Weight 


Weighted 


indicator 


Program 






rating 






rating 




Accounting 


1 


2 


2 


1 


1 


1 


3 


Carpentry 


1 


2 


2 


2 


1 


2 


4 


Child care 


2 


2 


4 


1 


1 


1 


5 


Farm equipment 


1 


2 


2 


1 


1 


1 


3 


mechanics 
















Fashion 


2 


2 


4 


2 


1 


2 


6 


merchandising 
















Nursing 


1 


2 


2 


2 


1 


2 


4 



The composite of the indicators could now be developed by simply adding 
the index numbers for a particular program and dividing by two to get an 
average. The results of this process are shown in Table 10. The indexed 
indicators could also be weighted, as was shown in Table 8, and the results 
would appear as in Table 11. The use of index numbers has the advantage of 
being able to better directly compare the different indicators for the same 
program (because the indexes have a common base of 100). 
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Table 9 

Index Indicators of Related Placement and Employer 
Satisfaction with Quality of Graduated Work 



Program 


Related 

placement 

index 




Employer 

satisfaction 

index 


Accounting 


105 




103 


Carpentry 


105 




99 


Child care 


59 




109 


Farm equipment 


108 




95 


mechanic 








Fashion merchandising 


81 




94 


Nursing (LPN) 


119 




94 


Statewide average 


100 




100 




Table 10 






Unweighted Composite 


Indexed Indicators of Related 


Placement and Employer Satisfaction with 




Quality i 


of Graduated 


Work 






Related 


Employer 


Composite 




placement 


satisfaction 


index 


Program 


index 


index 




Accounting 


105 


103 


104 


Carpentry 


105 


99 


102 


Child care 


59 


109 


84 


Farm equipment 


108 


95 


- 102 


mechanic 








Fashion merchandising 


81 


94 


88 


Nursing (LPN) 


119 


94 


107 



Table 11 

Weighted Composite Indexed Indicators of Related 
Placement and Employer Satisfaction with 
Quality of Graduated Work 



Related placement Employer satisfaction Composite 

Index Weight Weighted Index Weight Weighted index 

Program index ' index 

Accounting 105 2 210 103 1 103 104 

Carpentry 105 2 210 99 1 99 103 

Child care 59 2 118 109 1 109 76 

Farm equipment 108 2 216 95 1 95 104 
mechanics • - - 

Fashion 81 2 162 94 1 94 85 
merchandising 

Nursing 119 2 238 94 1 94 111 



Summary 



The focus of this chapter was essentially to try to learn about the 
process of developing vocational education indicators by actually "pushing 
ahead" with those indicators thought to be important and for which reasonably 
adequate data already existed. The steps in indicator development which were 
faced involved : ( 1 ) developing operational definitions , (2) displaying 
indicators, (3) setting benchmarks, and (4) combining indicators. From this 
experience the following summary points seem justified: 

• Adequate operational definitions can be constructed for some 
indicators which utilize existing information, 

• Displaying indicators involves considerations of both structure and 
format as a means to be understandable, focused and parsimonious, A 
structure and format which seemed to meet these criteria was proposed 
and illustrated, 

• Setting benchmarks for judging the adequacy of vocational education 
involves choosing appropriate comparisons, Appropriatness depends on 
purpose, process used, desired degree of responsiveness to external 
factors, and judgements of what is reasonable. Benchmarks become an 

integral part of indicator display and a basis for subsequent 

interpretation, 

• Combining indicators allows focus on the inter-relationship between 
indicators and summary of general condition across several indicators. 

As with the process of indicator verification discussed in the previous 
Chapter, each of the steps in indicator development must remain open to 
questioning and discussion. Far too little experience is available to make 
long lasting decisions about the exact way a vocational education indicator 
ought to develop at this point in time. 
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CHAPTER IV. 
FURTHER DEVELOPMENT OF 
INDICATORS WITHOUT ADEQUATE DATA 



Two of the indicators previously posed as important for vocational 
education in Minnesota were judged to be inadequate in data or, at least, in 
analysis of existing data to merit the direction of development described in 
the previous chapter. These indicators were: 

• Projected job openings in occupations related to training; and 

• Occupations for which program is designed to provide training. 

The strategy chosen for these indicators was analyses of existing data 
and attempts to improve this data. The purpose was to examine ways to make 
better use of what information was available or make recommendations for its 
improvement as a prerequisite to the indicator development illustrated in 
Chapter III. 

Projected Job Openings in Occupations Related to Training 

The necessary information to estimate the number ' of job openings for 
graduates of a particular vocational education program is not currently (and 
may never be) available in ideal form. Rather it is (and will likely remain 
to be) a matter of making the best use what is available and working to 
improve what is available. Major factors preventing quick and substantial 
improvement are cost, staff training and time and inter-agency coordination. 

Some Efforts At Projecting Job Openings 

The format worked out by the Minnesota Research and Development Center 
for interfacing labor market supply and demand information H981) represents 
an attempt to make best use of available information. An example of the 
application of this format is shown as Exhibit 16. The major constraint 
preventing the development of this format for other occupations is cost. A 
major limitation of the format is the lack of narrative analysis of what the 
information should mean for educational program planning or career selection. 



Exhibit 16. MRDC Format for Displaying Supply/Demand Information Related to 
a Specific Occupation 



MINNESOTA OCCUPATIONAL INFORMATION 

197S-1979 

DEMAND R ELATED DATA 

CURRENT DEMAND INDICATORS 

Current Employment 

Job Openings 

job orders received - Job Service 1 
jobs available - Job Service 1 
jobs open 30+ days - Job Service 1 
jobs available ^ 
new hires^ 

PROJECTED DEMAND INDICATORS 

Projected Employment (1985)** 
Projected Growth Rate (1978-1985) J 

Projected Job Openings (1978-1935) 3 
replacement job openings 
growth job openings 
total job openings 
annual average job openings 



SOURCES: 

1) Employment Service Automated 

Reporting Service 

2) Higher Education General Information Study 

3) Minnesota Employment Outlook to 1985 

4) Hew Hires Study 

5) Minnesota Vocational Follow-Up System 



OCCUPAT IONS : Accountants 



SUPPLY RELATED DAT A 
CURRENT SUPPLY INDICATORS . 
Current Employment 

Entrants from Education/Training Programs 

graduates from'AVTl's 5 

graduates from colleges and universities* 
Other Job Seekers 

Job applicants - Job Service 1 

unemployed AVTI grads* 

unemployed (general population)* 

COMPLEMENTARY INFORMATION 

Placement Rates of AVTI Grads 5 
related occupations 
closely related occupations 
broadly related occupations 

Placement Rates of AVTI Grads Available for Employment 
related occupations 
closely related occupations 
broadly related occupations 

Unemployment Rates 
AVTI grads 
UI applicants* 

Job openings' per applicant - Job Service 
Job placements per applicant - Job Service 
Median Monthly Wage - 1979 

♦unemployment information available at less detailed 
occupational levels from the Unemployment Insurance 
Program 



The Minnesota Occupational Information Coordinating Committee (MOICC), at 
the recommendation of the study report prepared by the MRDC, has begun providing 
narrative analysis of selected occupations as part of its newsletter entitled, 
News -Net . An example of this analysis is shown as Exhibit 17. The major 
constraint with this information is that only selected occupational areas are 
covered, one per issue of News-Net , a quarterly publication. 



ERIC 



• 82- 



88 



Exhibit 17 



MOICC Format for Narrative Analysis of Supply/Demand for an 

Occupational Area 



CAREER BRIEFS... 

INTRODUCING A NEW SERIES OF ARTICLES 

In response to the requests of readers expressed in our survey 
last Summer, as well as the direction of the MOICC Board, we are 
initiating a new series of articles analyzing supply and demand for 
selected occupations. 

Further, we will strive to use information sources which have 
not generally been presented heretofore. An "editorial board 11 
comprised of the most qualified representatives of labor market 
analysts and user and producer groups will review articles prior to 
publ i cation. 

Each issue will contain articles on a number of related 
occupational tions similar to the series, "Selected Legal 
Occupations," in this issue. We are actively seeking your feedback 
on the articles in regard to format, content and validity. Also, 
let us know which ocupations you would like to see profiled in 
future issues. Please call or write News-Net at your convenience. 

OUTLOOK FOR LAWYERS 

In 1978 approximately 487,000 people worked as lawyers in the 
United States. Approximately 70% were in private practice, while 
the balance worked in government and corporate envoronments. For 
information on the nature of the working conditions, education, 
etc., please consult the Occupational Outlook Handbook . 

The number of individuals employed as lawyers in Minnesota rose 
from 4,642 in 1970 to 8,130 in 1978 and will climb to 10,179 in 
1985, according to estimates based on the 1970 census. The number 
grew by 75 percent in the 1970's; however, that growth rate drops to 
25 percent between 1978-85. Compared to overall growth rates of 
14.4 percent for all workers, lawyers will increase faster than 
average (25%) during the 1978-85 period. 

Reports from the 3 local law school placement offices 
(University of Minnesota, William Mitchell and Hamline) show that 
the vast majority of recent law school graduates have otained a 
position in their chosen field within a year. More than 700 
graduates were reported by the three law schools in 1981. 
Typically, more than 85% of the graduates pass the bar examination 
and are employed each year. On the average it is projected that 692 
jobs will be open annually through 1985.* Competition for jobs is 
expected to remain intense throught the 80 's and lawyers with work 
experience will have the advantage as employers continue to be 
highly selective in hiring new laywers. 

According to the American Bar Association, lawyers in 
Minneapolis who had one year of experience were earning from $14,000 
to $28,000 in 1980. Average starting salaries varied from $17,520 
for graduates beginning with the State government to $17,800 for 
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those hired by business firms to $20,000 for those starting with the 
federal government in 1930. Local information on migration patterns 
and occupational mobility are not currently available for this 
analysis. 

The Bureau of Labor Statistics, U.S. Department of Labor 
(Bulletin 2052, September 1980) describes the outlook as follows, 
"The best prospects for establishing new practices will be in small 
towns and expanding suburbs, although this will remain a risky and 
expensive venture. An increasing proportion of law graduates will 
enter administrative and managerial positions for which legal 
training is an asset but not normally a requirement." 

It is apparent, while unsubstantiated, that the demand for 
lawyers, or more appropriately, legal services is a function of 
disposable income and federal social, environmental , and economic 
(primarily business) legislation. Examples include the Civil Rights 
Act, EEO, OSHA, etc. If there is a retrenchment on such legislation 
and'an effort to simplify (deregulate) business dealings (i.e., less 
anti-trust activity and less strict environmental laws), this could 
reduce the level of litigation activities and consequently, the 
demand for legal services. More detailed analysis is needed; 
however, if the demand for legal services declines, the outlook for 
all legal occupations diminishes as well. Users of this information 
should keep in mind this qualification during the early 1980's. 

★Minnesota Employment Outlook to 1985. 

OUTLOOK FOR PARALEGAL ASSISTANTS 

The outlook for paralegal assistants is less clear than that for 
lawyers or legal secretaries because it is a new and emerging occu- 
pation in some respects. The Occupational Outlook Handbook and most 
other information sources are silent about IT. Th~e Dictionary of 
Occu pational Titles (DOT) lists a major title, "paralegal assistant" 
with "law clerk" and "legal aide" as subtitles. The definition 
given describes all of them as providing assistance to lawyers in 
the research, analysis and preparation of legal matters. Typically, 
law clerks are law students or attorneys working for law firms or 
judges temporarily as part of their career development. 

Thus, the major problem in projecting the outlook for paralegal 
assistants is defining the occupation in measurable terms. 
Classifying, defining and measuring new occupations is a chronic 
problem for occupational and labor market analysts. Some of the job 
duties of paralgal assistants may also be performed by a law school 
graduate as well as by an experienced legal secretary either on a 
temporary or long-term basis. Certainly, the creation of paralegal 
jobs by a law firm or agency depends on its size and, to some 
extent, the nature of its business. Based on employer surveys admi- 
nistered through the Occupational Employment Statistics program (see 
legal secretaries) 1980 employment in these occupations was esti- 
mated as follows: Law Clerk — 550; Paralegal Personnel - 550. 

An article in the August 1981 issue of Monthly Labor Review 
published by the U.S. Department of Labor projects dramatic growth 
in the employment of paralegals — 132% between 1978 and 1990 in the 
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United States. The authors emphasize this is based on a conser- 
vative economic forecast. Based on the national trend the number of 
paralegals in Minnesota should more than double in this decade. If 
one includes law clerks and uses the 1980 estimate of 1,100 
employed, over 2,200 paralegals assistants should be employed by 1990. 

Training for paralegal/legal assistants is provided in both 2 
and 4 year programs at the following institutions: 
Four Year Programs 

- University of Minnesota, General College 

- Moorhead State College 

- Winona State College 

Two Year Programs 

- Inver Hills Community College 

- North Hennepin Community College 

- University of Minnesota, General College 

OUTLOOK FOR LEGAL SECRETARIES 

Of the 3.6 million people who worked as secretaries in 1978, 
162,000 worked in the legal field nationwide. In Minnesota, the 
number of legal secretaries rose very rapidly in the 1970's from 
1,558 to 2,760, or a 77% increase through 1978. A slower, yet 
strong, growth rate (54%) is predicted through 1985, or a level of 
4,236 legal secretaries by the mid-1980 f s. 

In 1980, the Occupational Employment Statistics Survey* 
estimated 3,400 legal secretaries were employed in Minnesota. An 
interpolation of 1985 occupational projections published in 
"Minnesota Employment Outlook to 1985"* yields an estimate of 3,200 
legal secretaries employed in 1980. Occupational data from the 1980 
Census will not be available until mid-1982. 

The Minnesota Department of Economic Security, in its 1980 "New 
Hires Study, 11 indicates that 356 legal secretaries had been hired in 
the State between December 1979 and November 1980. Interestingly 
enough, annual average job openings for the period 1978-85 are 
estimated at 357 annually. 

'A total of 250 graduates of legal secretary courses were 
reported by public post-secondary vocational school follow-up 
surveys with 76% employed in related fields within a year of 
graduation. A total of thirteen, or 5.2% of the graduates were 
available for work but unemployed one year after graduation. 
Information on the numbers graduating from the State's community 
colleges or private post-secondary "business" schools is not now 
available; thus, estimates of total supply to compare to projected 
annual openings (357) cannot be prepared at this time. 

First year salaries ranged from $300 per month to $1200 per 
month with a median wage of $700-800 per month. This data was also 
collected by the postsecondary vocational institutions one year 
after graduation, between July 1980 and June 1981. 

Legal secretary programs were available at 24 of the 33 AVTTs 
around the State. For program descriptions, locations, etx., please 
call 296-6481 within the Twin Cities metropolitan area. Those 
living outside the 7-county metropolitan area should call the toll 
free number 1-800-652-9021. 
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Many of the public community colleges offer legal secretary 
programs or general secretarial with legal specializations. In 
addition, 7 private "business 11 schools offer legal secretary 
programs. • 

High school students and counselors may access the same 
information via Minnesota Career Information System (MC1S) 
terminals. 

^Prepared by the Minnesota Department of Economic Security. 



Source : Minnesota Occupational Information Coordinating Committee . 
Career Briefs. . .Introducing A New Series of Articles. News -Net , Vol. 
Ill, No. 1, January, 1982, p. 4-5. 

It seems the most appropriate, one source of information on job openings 
available from the Minnesota Department of Economic Security is that 
entitled, Minnesota Employment Outlook to 1985 . The most resent edition is 
January, 1981. A special edition for the Minneapolis/St. Paul area was 
published in March, 1982. A sample of the format of data in this publication 
is shown in Exhibit 18. While this publication provides a good overview of 
the projected occupational requirements, the difficulty for use in vocational 
education planning is in matching vocational education program titles and 
occupational titles. For some programs there is a close match and for others 
the match is impossible to make with any accuracy. Some attempts have been 
made by the MRDC to relate programs and occupation as shown in the format 
developed for the labor market supply/demand tables in the Minnesota State 
Plan for Vocational Education (See Exhibit 19). As can be seen there is much 
left to be done in identifying occupations related to each program and 
relating the classification systems for programs and occupations. 

During the course of the project described in this, report an attempt was 
made to become familiar with new efforts being made to improve labor market 
demand and supply information in Minnesota. Two particularily relevant 
efforts were identified under the direction of Rudy Pinola in the Research 
and Planning Section of the Minnesota Department of Economic Security. 

One of these efforts focused on trying to define what is meant by a 
"shortage" and "surplus 11 in the labor market. The issues under investigation 
were: (1) the actual ratios of job openings to job applicants which were 
considered to be a shortage, surplus or balance market for a selected sample 
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Exhibit 18 

Sample of Information from Minnesota Employment Outlook to 1985 



ELECTRICIANS 



Employment 1978 10,960 

Projected 1985 Employment. 12,300 

Percent Growth 1978-1985 12.2 

Total Openings 1978-1985 3,250 

Expansions 1,340 

Replacements 1,910 



Major Industrial Employers 1978 - Manufacturing; Construction, Mining. 

Job Replacements - Most training authorities recommend completion of a four 
year apprenticeship program as the best way to learn the tr.ade. Many workers 
learn informally by working as helpers for many years. Many helpers obtain 
additional knowledge through trade school or the Armed Forces. 

EXCAVATING, G RAPING MACHINE OPERATORS 



Employment 1578 7,660 

Projected 1985 Employment 8,510 

Percent Growth 1978-1985 11.1 

Total Openings 1978-1985 2,500 

Expansions 850 

Replacements 1,650 



Major Industrial Employers 1978 - Construction; Mining. 

Job Replacements - Many construction machinery operators learn their skills 
on the job, but most training authorities recommend completion of a three 
year apprenticeship as the best way to learn the job. Since apprentices 
learn to operate a variety of machines, they have better job opportunities. 



Source: Minnesota Department, of Economic Security. Minnesota Employment 
Outlook to 1985 / St. Paul, MN: Minnesota Department of Economic 
Security, January, 1981, p. 61. . 

of occupations, (2) the difference in definition as viewed by various groups 
such as employers and job applicants, and (3) other factors which should be 
considered in defining a shortage and surplus. These definitions would seem 
critical to decisions about how many people to train in vocational education 
programs. 

A second study being conducted by the Department of Economic Security 
focused on trying to define homogeneous occupational groupings. That is, 
occupational titles or categories which were very similar in acceptable 
qualifications for job applicants. The point being examined was that many 
occupational titles may be too "general" for meaningful use in relating labor 
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Exhibit 19. Sample of Supply/Demand Information in Minnesota State Plan for 
Vocational Education. 
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Source: Minnesota State Department of Education, Minnesota State Plan for 
Vocational-Technical Education . St. Paul, MN: Minnesota State 
Department of Education, 1981, p. 

supply and demand. For example, the occupational title of "carpenter" 
contains a wide array of sub-occupational categories which vary considerably 
in acceptable applicant qualifications (i.e., from being physically strong 
and not having any particular training to having specific supervisory 
personal characteristics, specific training, and several years of work 
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experience). Of relevance to vocational education is that its programs may 
prepare acceptable job applicants for only a certain range of these sub- 
occupations within carpentry and there may or may not be demand for these 
types of applicants—none of which would be evident from just examining labor 
demand information for the broad occupational title of carpenter. Findings 
from these two studies will be available in the next year or so, 

A Proxy Measure for Projected Job Openings 

In view of this context, it is proposed that a proxy measure for job 
openings information be used for the process of prioritizing vocational 
education programs in terms of labor market demand or in conjunction with 
other available labor market demand information. The proposal is that the 
measure to be used is "percent of program graduates placed in related 
occupations," 

Advantages and disadvantages , The advantage of using percent related 
placement is that it gives a very specific measure of the ability of 
graduates to obtain jobs for which they were prepared. It is specific to the 
training provided and labor market encountered by the graduates. The 
assumption made is that if graduates can get related jobs, labor market 
demand exists and/or graduatres can successfully compete with other sources 
of labor supply for these openings. Further, information on percent related 
placement is readily available from the follow-up system for the past several 
years. 

The disadvantage of using precent related placement as a measure of 
occupational demand is that there may be reasons other than low demand which 
account for low related placement. For example, the graduates may not have 
received the "right" training (i.e., program was appropriate but curriculum 
inappropriate) or they may- not have learned (i.e., program and curriculum 
appropriate but little learning took place). The information is always at 
least a year old since it is derived from former graduates approximately a 
year after leaving school. Also critical as a limitation is that this type 
of information is not availabe for programs that have not been operated at 
least once. This latter limitation is especially relevant to decisions about 
starting new programs; however, it may be possible to use information from 
closely related programs or programs operated by other agencies or 
institutions. -89- 
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Perhaps these limitations are somewhat compensated for by the information 1 s 
specificity to a particular training program and geographic area, also for 
some occupations, the demand does not change radically from year to year. 

Typical i nformation . The information (simulated) for a sample of 
programs for the percent related placement might appear as in Table 12. 

Table 12 

Information (simulated) About Percent 
Related Placement for a Sample of Programs 



Program 



Related 
placement (%) 



Program 


A 


80 


Program 


B 


75 


Program 


C 


60 


Program 


D 


90 


Program 


E 


40 



Process of assigning priorities . The assignment of vocational education 
program priority based on the above indicator could be done very generally or 
in greater specificity through some form of indexing. For example, a general 
rating process is illustrated in Figure 22. The information on related 
placement could be placed on a line and programs arbitrarily assigned a high, 
middle or low rating based on^ their percent related placement. 

5 100? 



Percent 
related 
placement 



80 

60 

40 
20 
0 



D 
A 
B 



High 



C 
E 



Middle 



Low 



Figure 22. General rating of programs in terms of labor demand using percent 
related placement as a proxy measure. . " ft-y 
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A more specific indicator of program priority in terms of labor. market 
demand could be developed by converting the information on related placement 
to index- numbers. For example, related placements could be indexes such that 
100 is set as the average percent related placements across all programs. 
For the five sample programs, this average is 71 percent [(80 +75+60+90 
+ 50)/(5)J. Given that related placement of 71 percent is indexed at 100, 
then a related placement of 60 percent is equal to ( 100) (60)/?1 =85. If this 
process of indexing were used, the results would appear as in Table 13. 

The index resulting from either the general or this specific rating of 
programs in terms of labor market demand using the proxy of percent related 
placement could then be considered together with indicators representing 
other important factors (e.g., employer satisfaction, cost) in planning, 
reviewing and providing public information about vocational education. 



Table 13 

Results of Using Indexing to Derive a More Specific Rating 
of Labor Market Demand for a Program Using the Proxy 
Measure of Percent Related Placement 



Program Index of related 
placement 



Program A 113 

Program B 106 

Program C 85 

Program D 127 

Program E 70 

Note: 71% = 100 for related placement. — - — — - 

An Underlying Principle in Considering Projected Job Openings Information as 
an Indicator 

Underlying the development of an indicator of job openings or labor 
market demand more generally is the proposed notion that vocational education 
is first responsive to people and then to the labor market. The idea is that 
vocational education program outputs should be responsive to labor market 
demand in relative terms rather than absolute terms. For example, let f s 
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assume vocational education faced a very simple world with only three 
possible occupations and a program designed for each occupation. Further, 
assume there are only 100 job openings and three possible situations in terms 
of number of people wanting training — they are 50 people want training, 100 
people want training, and 200 people want training. The principle being 
advocated is that using only the factor of labor market demand, vocational 
education' should prepare all of the people wanting training in each 
situation. (There may well be other factors such as a budget constraint 
which would prevent this from happening). 

However, in each situation the number of persons prepared by each of the 
programs would change. This number is shown in Table 14. The point is that 
the proportions of persons in each program would remain the same across the 
three situations — vocational education is responsive in relative terms. 
But, as shown for the situation where there is 200 who want training and only 
'100 job openings, vocational education is not responsive in absolute terms. 
Rather, all people who want training are served but in proportions relative 
to labor market demand for each occupation. In this example, occupations are 
treated as being of equal importance — this could be changed if a measure 
such as wages were used to indicate the relative importance of each 
occupation. 



Table 14 

Response of Vocational Education to Various 
Labor Market Demand Situations 



Occupation/program 


Labor market 
demand 


Number 
50 want 
training 


in each program 

100 want 200 want 

training training 


A 


20 


10 


20 


40 


B 


50 


25 


50 


100 


C 


30 


15 


30 


60 


Total 


100 


50 


100 


200 
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Occupations for Which Programs Are Designed 
to Provide Training 

Before one is really able to derive information about the percent of 
graduates entering related occupations or the responsiveness of vocational 
education programs to labor market demand, it is necessary to define which 
programs "relate" to which occupations. This has not been done in Minnesota. 
That this indicator is fraught with many issues is evident from the analysis 
presented in Chapter II. 

The relationship between a particular vocational education program and 
one or more occupations could take any of the forms shown in Figure 23. A 
program could be related to a single occupation (e.g., practical nursing) or 
to several occupations (e.g., auto mechanic). Also, a single occupation 
could be related to several programs (e.g., hardware store sales person). 

Vocational education 

program occupation 




Figure 23. Relationship of vocational education program to specific 
occupations. 

Several efforts have been made to relate vocational education programs 
to occupations and vice-versa. For example, Table 15 shows the results of 
work by the MRDC in 1977 indicating the occupations entered by graduates of 
the post secondary carpentry programs. Table 16 shows the programs having 
graduates who entered the carpentry occupation. Some of this process has 
been incorporated into the labor market supply /demand tables in the Minnesota 
State Plan for Vocational Education. 

-93- 



93 



Table 15 

Proportion of Carpentry Graduates Entering Various Occupations (1977) 



Sales Managers and Department Heads Retail Training 01 

Sales Managers Except Retail Trade «... .01 

Salesmen and Sales Nec 02 

Cabinetmakers . . 02 

Carpenters 65 

Carpenter Apprentices 11 

Compositors and Typesetters 01 

Excav., Grdg., Rd. Mchn. Operator Exc. Boozer 01 

Automobile Body Repairmen 01 

Miscellaneous Mechanics and Repairmen , 02 

Pressmen Plate Printers, Printing 01 

Mine Operatives Nec. 01 

Machine Operatives, Miscellaneous Specified 03 

Deliverymen and Routemen 02 

Gardeners, Groundskeepers Exc. Farm 01 

Miscellaneous Laborers 01 

Janitors and Sextons 04 



1.00 



Table 16 

Proportion of Graduates of Various Programs Entering Carpenters Occupation 

(1977) 



Carpentry 72 

Mechanical Draft and Design , 01 

Architectural Draft and Design .* , .04 

Machine Shop 01 

Welding 02 

Chefs, Cooks, Food Preparation • .02 

Cabinetmaking , .02 

Marine and Small Engine Mechanics... 01 

Bricklayer , .01 

Refrigerator, Air Conditioning, Heat, Appliance Repair .01 

Agri Technology " .01 

Accounting 01 

Farm Management , e 01 

Telephone Communication Tech 01 

Conservation and Forestry 0 01 

Photographic Technology 01 

Construction Worker 01 

General Sales 03 

Business Management 01 



1.00 
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Exhibit 20. Crosswalk of Vocational Education Programs and Occupations 
Developed by NOICC. 



PROGRAM. U6.0U06 GLAZING. 



AN INSTRUCTIONAL PROGRAM THAT PREPARES INDIVIDUALS TO PREPARE, FIT, AND INSTALL 
GLASS IN STRUCTURAL OPENINGS SUCH AS WINDOWS, DOORS, AND PARTITIONS OR IN DISPLAY CASES 
. AND TABLE TOPS. • • 



. DICTIONARY OF OCCUPATIONAL TITLES 
«4TH CODE 

775 60U-022 GLASS CUTTER 

775 60M-026 GLASS FINISHER 

775 607-018 GLASS-CUTTER HELPER 

779 381-010 GLAZIER, STAINED GLASS 

779 68H-022 GLASS CUTTER, OVAL OR CIRCULAR 

865 381-010 GLAZIER " 

865 381-014 GLAZIER APPRENTICE 

865 48M-010 SAFETY-GLASS INSTALLER 

865 68M-01U GLASS INSTALLER 

865 63M-010 GLAZIER, METAL FURNITURE 

865 68M-022 REFREGERATOR GLAZIER 
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Source: National Occupational Information Coordinating Committee. 

Vocational Preparation and Occupations . Washington, D.C. : 
National Occupational Information Coordinating Committee, 1982. 



In addition a national effort has been made by the National Occupational 
Information Coordinating Committee (NOICC) (1982) to develop a "crosswalk" 
between vocational education programs and various occupational classification 
schemes. A sample of their results is shown as Exhibit 20. A computer tape 
now. exists which contains this information for upwards of 200 vocational 
education programs. Additionally NOICC has recommended and developed a 
process for deriving "units of analysis" which are grouping of occupations 
that are reasonable and practical to think about when planning educational 
programs (National Occupational Information Coordinating Committee, 1982). 

Most recently, the MRDC has been examining the use of student's 
judgements of relatedness to identifying occupations related to a particular 
vocatinal education program. Former students - are asked through the follow-up 
system if their present jobs are related, somewhat related, or unrelated to 
the program from which they graduated. A sample of this information for the 
farm equipment mechanics program is shown as Table 17. An analyses of this 
kind for all programs for several years combined with the crosswalk 
information and unit of analysis concept provided by the National 
Occupational Information Coordinating Committee could result in a consistent 
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and agreed upon set of related occupations for each vocational education 
program. Lack of consistent and stable definition of occupations related to 
each program category will constrain the accurate interpretation of any 
relatedness information and use of labor market demand information in 
planning reviewing and providing public information about vocational 
education in Minnesota „ 



Table 17 

Using Students' Judgements to Indicate Related 
Occupations: Farm Equipment Mechanics Program, 1974-1976 



Occupation entered 



Number Percent indicating 
entering related 



Not specified mechanics and repairs 
Heavy equipment operators 
Miscellaneous mechanics and repairs 
Delivery and route workers 
Stock clerks and store keepers 
Supervisors, nec 
Assemblers 

Garage workers and gas station attendants 
Farm implement mechanics 
Farm supervisors 
Farmers 

Managers and administrators, nec 
Sales clerks, retail trade 
Welders and flame cutters 
Sales persons , retail trade 
Carpenters 

Automobile mechanics 
Mill wrights 

Machine operatives, not specified 

Motor workers: mine, factory, logging camps 

Truck drivers 

Construction laborers, exc. carpenters' helpers 
Miscellaneous laborers 



11 
7 
3 
2 
1 
1 
1 
1 

30 
14 
4 
3 
2 
2 



100% 
100 
100 
100 
100 
100 
100 
100 
90 
86 
75 
66 
50 
50 
0 
0 
0 
0 
0 
0 
0 
0 
0 



Total 



90 



81 
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Summary 



The focus of this chapter was on further development of those indicators 
which were judged to be important but did not have, adequate data readily 
available for pursuing the developmental steps described in Chapter III. 
Instead, an attempt was made to analyze existing information toward making 
recommendations for its use and improvement* The summary points which seem 
justified are as follows: 

• While much labor market information is available, none in sufficiently 
specific complete, 'or compatible with that available for vocational 
education programs for use in indicator development. 

• It may be reasonable to use proxy measures of labor market demand 
derived from follow-up information on graduates of vocational 
education programs as a "stop-gap" mechanism in combination with 
existing labor market demand information until more adequate 
information is available. 

• Little progress can be made in using labor market information to 
develop a relevant vocational education indicator until a consistent 
and stable set '>f occupations related to each program is specified. 
The use of existing "crosswalks" between occupations and programs, the 
concept of "units of analysis" and existing follow-up information on 
occupations judged "related" by students after good possibility to 
resolve the relatedness issue at least in terms of the best means to 
use for this purpose. 
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CHAPTER V. 
NEXT STEPS IN INDICATOR DEVELOPMENT 

The purpose of this chapter is to look across the work reported on in 
previous chapters in terms of the general problem area of further developing 
adequate vocational education indicators in Minnesota. Each of the above 
chapters already has a summary section describing major conclusions of each 
separate effort. 

Looking then across the chapters, the following recommendations seem in 
order: 

• The process of indicator verification and development must be 
continous . The results of the efforts described in this report 
suggest progress has been made but that there are many ways to improve 
what has been done. New indicators need to be considered especially 
in view of current social and economic issues facing the nation and 
more particularily Minnesota. Indicators must be linked to an overall 
purpose of vocational education as might be formulated in a mission 
statement. A conceptual framework is needed to related indicators to 
one another and insure comprehensive coverage of the phenomenon 
labeled vocational education. Each indicator must be critically 
examined in terms of the ends it is directed toward, what it really 
means and how it is to be measured. Several other verification 
strategies suggested, but not tried, in this project need to be 
examined for their use in assisting responsible individuals to feel 
more confident about the indicators eventually selected (e.g., 
obtaining views of groups such as parents, teachers, students, school 
board members, employers, and legislators; examining indicators from a 
futurist's perspective; linking indicators to the leading economic 
indicators for Minnesota; examining the history of development of 
accepted indicators such as the unemployment rate for lessons in 
indicator development). For certain indicators, data needs to be 
improved substantially either by making new and creative use of 
available data or gathering new forms of data. Other indicators, with 
adequate quality of data, need refinment in operational definitions, 
display structure and format and in setting appropriate benchmarks. 

• The need for both reflective thought and experience are important to 
indicator development . There is a symbiotic or "hybrid effect" of 
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being en&aged in rather theoretical discussions about what would be 
ideal indicators and, at the same time, to be engaged in actually 
constructing indicators from what is available. Reflective thought 
provided direction, criteria and coherence for judging experience; 
experience provides relevance, tangibility and a sharpness of 
questioning. Discussion and thought could and should go on forever 
but at least as much likely will be learned by doing— actual 
development and use of indicators using best judgement and what 
information is availabls. The tension between "not being sure" yet 
"having to do" is productive to indicator improvement, 
• Some of the indicators must be pushed to the stage of actual use in 
planning, review and providing public information . The real test of 
the viability of the indicator concept will only come with use over a 
period of time in real life situations. Only here will the 
limitations of indicators, especially in terms of consequences, be 
most clearly exposed for consideration. And, if the indicators are 
used, the rationale for their continuance and improvement will be 
provided. A major next step in indicator development, following the 
step of setting benchmarks, will be the interpretation of what an 
indicator is implying for vocational education in general or for 
specific programs— what does it mean. Also during use, aspects of 
indicator feasibility such as staff training needs, scheduling 
deadlines, degree of accuracy, and operating costs will be more 
readily apparent. 

# Involvement of those using and affected by use of the indicato rs is a 
necessary part of indicator development . The Work Group used in this 
project provided important insight and perspective to the activities 
which were a part of this project. Some of this advice is most likely 
"to only come from individuals who would have staff responsibility to 
actually calculate and interpret indicators, make decisions based on 
the interpretation of indicators as a major input or bear the 
consequences of decisions made using indicators. With shared 
responsibility and involvement during indicator development, these 
same individuals gain an understanding and committment to indicator 
use that can be, at most, approached superficially on a post hoc 
basis. Involvement of affected individuals adds a discipline and 
seriousness to the effort that would be difficult to "spark" in other 
ways. 
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Tentative Agenda for Work Group on Vocational Education Indicators 
(10:30 a.m. - 12:30 p.m., June 15, 1982) 
Meeting No. 1 

1. Review purpose of research project and work group. (15 minutes) 



Sharing of experience with development or use of "indicators idea" by 
work group members. (30 minutes) 



Examine progress in identification of vocational education indicators 
for Minnesota. (30 minutes) 



4. Discussion of "next-steps" in indicator identification and development, 
(40 minutes) 



5. Plans for next meeting. (5 minutes) 



Goal for Meeting 

Concensus on reasonable next steps in the development of vocational education 
indicators for Minnesota. 
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Tentative Agenda for Work Group on Vocational Education Indicators 
(1:00 - 3:30 p.m., July 14, 1982) 
Meeting No. 2 



1 . Review discussion at last meeting and tentative decision as to direction 
for Vocational Education Indicators Project. (15 minutes) 



2. Further verification of vocational education indicators in terms of: 

— Vocational education as responsive to current social and economic 
needs. 

— Vocational education as a part of education . (45 minutes) 



3. Operational definitions for those indicators with adequate data: 

— Employer's satisfaction with quality of graduate work. 

— Graduates employed. 

— Graduates employed in occupations related to program. 

— Program cost. (40 minutes) 



4. Developing better data for those indicators with inadequate data: 

— Projected job openings in occupations related to program. (Briefing on 
new efforts by the Minnesota Department of Economic Security). 

— Occupations for which program is designed to provide training. (40 
minutes) 



5. Set time and focus of next meeting. (10 minutes) 
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Tentative Agenda for Work Group on Vocational Education Indicators 



(9:00 - 11:30, August 10, 1982) 
Meeting No. 3 



1. Review of discussion at last meeting and agenda for this meeting, (15 
minutes) 



2. Discussion of paper shared at last meeting by Joe Malinski. (15 minutes 

3. Operational definitions for those indicators with adequate data 

— Criteria for operational definitions 

— Criteria for displaying indicators 

— Setting benchmarks 
(40 minutes) 



4. Further verification of vocational education indicators 

— Your educational philosophy 

— Alternative visions of vocational education 

— Implications for important indicators 
(55 minutes) 



5. Advice on future steps for indicator development, (20 minutes) 

6. Closing remarks for work group. (5 minutes) 
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User's Guide to Accompany 



MINNESOTA ANALYSIS 

OF BELIEFS 
Tn EDUCATION 




BELIEFS ARE THE RESULT OF EXPERIENCE... SEE INSIDE 



Minnesota Department of Education 
Division of Instruction 
Elementary and Secondary Education Section (ESES) 
650 Gapitol Square Building 
St. Paul, Minnesota 55101 
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The Blind Men and the Elephant 



John G. Saxe 



It was six men of Indostan 

To learning much inclined, 
Who went to see the Dephant 

(Though all of them were blind), 
That each by observation 

Might satisfy his mind. 

The First approached the Elephant, 

And happening to fall 
Against his broad and sturdy side, 

At once began to bawl: 
"God bless me! but the Elephant 

Is very like a wall!" 

The Second, feeling of the tusk, 
Cried, "Ho! what have we here 

So very round and smooth and sharp? 
To me 'tis mighty clear 

This wonder of an Elephant 
Is very like a spear!" 

The Third approached the animal, 

And happening to take 
The squirming trunk within his hands, 

Thus boldly up and spake: 
"I see," quoth he, "the Elephant 

Is very like a snake!" 

The Fourth reached out his eager hand, 

And felt about the knee. 
"What most this wondrous beast is like 

Is mighty plain," quoth he; 
"Tis clear enough the Elephant 

Is very like a tree!" 

The Fifth, who chanced to touch the ear 
Said, "E'en the blindest man 

Can tell what this resembles most; 
Deny the fact who can, 

This marvel of an Elephant 
Is very like a fan!" 

The Sixth no sooner had begun 

About the beast to grope. 
Than, seizing on the swinging tail 

That fell witiiin his scope, 
"I see," quoth he. "the Elephant 

Is very like a rope!" 

And so these men of Indostan 

Disputed loud and long, 
Each in his own opinion 

Exceeding stiff and strong. 
Though each was partly in the right 

And all were in the wrong! 




Minnesota Analysis of Beliefs in Education: A 

Description 



The Minnesota Analysis of Beliefs in Education (MABE) is an information-gathering tool designed to provide 
technical assistance to schools and school districts. MABE is a low-cost, standardized way to discern the 
belief systems of groups of people, such as staff members, parents, and other community members. MABE 
provides an inventory of beliefs and assumptions. MABE functions neither to promote any particular set of 
beliefs or assumptions nor to direct educational decisions. Rather, it is an inventory for the collection of data 
— the use of which enables educational decision-making to be more thorough and productive. 

Rationale 

In the face ol conflict we often neglect to recognize that conflict is rooted in differences of beliefs and 
assumptions people hold. Similarly, efforts to implement particular educational programs or practices are 
frequently frustrated because of the conflict resulting from these same differences of beliefs and 
assumptions. 

On the other hand, educators are frequently criticized because of their apparent inability or failure to relate 
beliefs/assumptions and educational practices. Venable has succinctly described this state: 

Consistency between beliefs and the development of practices growing out of these beliefs are all too 
rare among those of us who are concerned with education. 

Lacking this consistency, we may find ourselves giving lip service to two or more conflicting views, or 
professing one belief while our behavicr evidences another, or — worst of all — allowing our behavior to 
be determined by forces that bear no relationship to any particular set of principles. 

(Venable, T.C., Philosophical Foundations oLthe Curriculum, Chicago: Rand McNally and Co., 1967, 
pages 9-10.) 

The MABE inventory allows individuals and groups to review the systematic beliefs from five traditional 
schools of philosophy and to see where their own beliefs and assumptions tend to cluster. Such informa- 
tion helps the school or school district to recognize the consistency or lack of consistency among 
generally held beliefs and the existing school programs and practices. It also helps predict potential dif- 
ficulties surrounding the implementation of new programs or practices to the extent that the relationship 
between the beliefs of the various publics involved and the new programs or practices can be identified 
and analyzed. In all of this, the MABE inventory is a possible resource for gathering information which 
should assist in determining the causes of educational conflict, actual or potential. 

Schools or school districts typically attempt to develop an educational philosophy and goals that are reflec- 
tive of the community and staff. MABE holds the promise of providing basic information upon which to 
base philosophy and goals. So, whether the desire is to understand the basic belief orientation of a group 
of education professionals or of the community, this inventory can be a valuable aid. 

In short, MABE helps assess the general philosophic positions of selected groups of individuals. The infor- 
mation gathered from the use of the inventory ought to be of value in setting educational goals, resolving 
conflict and providing significant data for decision-making. 

Summary Information Obtained from MABE 

MABE covers three broad areas: Human nature, knowledge, and values. It consists of 90 items which are writ- 
ten in nontechnical language. The content of the items relates to the beliefs of five different philosophical ap- 
proaches. 



An individual's response to the 90-item inventory can be collapsed onto a graph which shows the set of 
beliefs the individual tends to subscribe to from the traditional schools of philosophy, and the set of beliefs 
the individual does not agree with. The responses, as graphed, show how ideas cluster around a particular 
philosophical position or spread across several philosophies. 

Composite scores can be obtained for an entire group of people and these, too, can be graphed. In this way, 
comparisons can be made across groups of people with respect to how closely they are likely to agree about 
what human beings are like, how they learn, what they should be taught and how they acquire values. 
It is important to reiterate that the individual or group profiles gathered as a result of the use of the MABE in- 
ventory do not require any particular action. Information obiained from MABE does provide data about the 
assumptions/beliefs held by the participants— assumptions/beliefs that affect a point of view regarding 
educational programs and practices. 



Philosophic Positions 

In order to facilitate the use of MABE, brief descriptions of the five philosophic positions are presented. While 
it is not possible to represent these five schools of philosophy adequately with a few statements, the general 
descriptions provided are, presented as a means of underscoring the differences that exist among the 
philosophic approaches represented. Users of MABE are reminded that many excellent texts are available 
which detail these philosophic positions. 

Idealism 

This very early philosophic position maintains the dominance of the world of ideas over the world of sense 
reality. Therefore, emphasis is given to mental discipline, the development of ideas and concepts, and the 
search for universal truths. 

It is the work of education, according to the Idealist, to assist students in discovering the ideal truths that exist. 
This is commonly accomplished through such academic fields as history, mathematics, and literature. 

The Idealist believes that absolute and unchanging values exist. These values are timeless and are found in 
each society and throughout history. Through the use of reason and logic, the human person can support 
belief in such values. 

Essentialism 

Essentialism takes its name from the fact that there exist certain essential knowledge and beliefs. This 
knowledge and these beliefs are knowable. 

Typically the Essentialist position unites the main beliefs of two other philosophical schools of thought: 
Idealism and Realism. Realism is a philosophic position which advocates the. objective reality of our world. 

Essentialism often has been linked with traditional religious beliefs. This results primarily from the Essen- 
tialist view of the student as a fugitive from learning— an assumption which corresponds to the doctrine of 
original sin or the sin of Adam and Eve. In other words. human nature is thought to be inclined toward "evil." 
The instructor must practice consistent discipline in order to assist the student in developing the self dis- 
cipline required for learning and a worthwhile life. 

Essentialists view the teacher as a master of learning. There is a knowable body of knowledge that teachers 
share with students in leading them to become well-educated adults. 

As with the Idealist position, Essentialism acknowledges the existence of absolute and unchanging values. 
Pragmatism 

This philosophic position is associated with the later progressive movement in American education. 
Pragmatists contend that truth is found in that which works. An idea is tested by examining its consequences; 
the degree to which the consequences are acceptable is the degree to which truth exists. 

The Pragmatist does not simply advocate something that is seen to be useful for an individual. Rather, truth is 
that which helps us move toward what is progressively good for our community or society. 

In other words, truth is relative and will vary according to time and place. Therefore, for the Pragmatist there 
are no absolute or changeless truths or values, other than perhaps Pragmatism itself. 

Reality, for the Pragmatist is viewed as one's experiences. One's experience in trying to know or solve 
problems accounts for ones relationship with the real world. This relationship is viewed as a process. In fact, 
the Pragmatist position is said to be process-oriented. 



In education, the Pragmatic point of view encourages students to learn through experiences of the real world, 
both physical and social, and to solve problems associated with those experiences. 

Existentialism 

The Existential position focuses on the individual. Each person is thought to be unique and, therefore, efforts 
to impose group values upon the individual are not acceptable. 

In order to cultivate and maintain the unique character of the individual, a high degree of personal freedom is 
required. The person is not in search of some ideal or absolute truth. The individual strives to come to know 
his/her own being and person. For it is in knowing who the individual is that he/she can determine what 
reality is. 

Similarly, each individual, given personal freedom, determines the values that are important It is obvious that 
this philosophic position is subjective in its attention to the individual and to the individual's personal deter- 
mination of truth and values. 

In education, the Existentialist assists students to create their own unique existence and being, and to 
recognize and practice personal freedom. Through personally selected experiences, the individual grows in 
self knowledge and achieves self-fulfillment 

Behaviorism 

Two separable aspects of a system which is derived from psychology are: Behaviorism as a philosophic posi- 
tion and behavioral engineering as a technology. The philosophic position asserts that the human person is 
conditioned in all phases of life to respond and react in certain ways. Knowing this, it becomes clear that the 
human behavior is not free, but rather is determined by environmental stimuli. In addition to being controlled 
by such obvious stimuli as those provided in our physical environment, we are also controlled by the stimuli 
managed by other human' beings. 

A stimulus can be either positive, as in a reward, or negative, as in punishment It is believed that human 
behavior can be modified by establishing the "right" set of conditions. Certain members of society, usually 
those associated with the scientific/technological community, have the responsibility to identify the condi- 
tions which change behavior for the good of society and the individual. 

Behaviorists generally are silent on the topic of values. That behavior which results in positively reinforcing 
stimuli is seen as good. 

in education, the Behaviorist is keenly interested in maintaining the right set of conditions for producing 
learning, e.g., a learning environment in which correct behavior is followed by positive consequences. Stating 
specific behavioral outcomes of learning in some quantifiable way and modifying student behavior toward 
learning reflect the usual concerns and activities of those who subscribe to Behaviorism. 



Preparing to Administer MABE 

The person who administers MABE should become thoroughly familiar with the user's guide before ad- 
ministering MABE to a group. In addition, the administrator should actually take the inventory before giving it 
to others. This item-by-item preparation will allow the person administering MABE to share the experience of 
the participants. 

Materials. 

Initially give out to each participant the following: 
1 copy of the MABE inventory 
1 pencil 

1 name tag (optional) 
After the participant has taken the inventory give out: 

1 copy of MABE Scoring Sheet and Individual Profile 

1 MABE Description of Philosophical Positions 

1 lally sheet for total scores of individual (optional). 
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Administration. There are no formal time limits set for MABE. Plan a time block of about 30 minutes to ad- 
minister MABE to a group. Instructions to persons taking MABE sfiould be to respond to the items on this 
forced-choice inventory by deciding whether you mostly agree or mostly disagree with each item. 

The administrator should give no interpretation of the items to the participants. Often the respondent's first 
impression is a more reliable response than an intensive analysis of the item. It is, however, the nature of 
philosophical reflection to ponder and to analyze. No participant should be discouraged from doing this or 
rushed into a hasty decision if this is not the person's style of responding. 

Tell the participants that after they have completed the inventory, they will be given a scoring sheet to deter- 
mine for themselves into which of the five philosophical schools their responses tend to fall. They will also be 
given a sheet which briefly describes each of the five philosophical schools used in the inventory. They can 
use the time while they wait for others to complete the inventory to study their own responses in the light of 
the descriptions. 

Scoring. To score his/her own responses, the participant should find the items the person mostly agreed 
with and circle those numbers on the scoring sheet. The items are listed on the scoring sheet under the 
philosophical school to which they belong. For example, in looking at responses to the first five items, if num- 
bers 1 , 3 and 4 were agreed with, the person would circle those numbers which are found under the schools 
of Behaviorism, Idealism, and Essentialism. The numbers 2 and 5 would be without circles since the respon- 
dent did not agree with the positions of the Pragmatism and Existentialism schools. 

A sample scoring sheet for one individual is shown in this user's guide. Total scores for each of the five 
schools are found by adding the number of circled items. The total scores can then be plotted on the graph 
at the bottom of the scoring sheet. 

Individual Analysis . Point out to the participants that the scoring sheet shows numbers in three separate 
clusters. The top third of the numbers are for items in the inventory which ask about human nature. The mid- 
dle third of the numbers on the scoring sheet are items from the inventory that concern knowledge. The last 
third of the numbers are responses about moral and aesthetic values. An individual can look at his/her 
responses within schools, but it is also possible to analyze one's responses for the three areas of human 
nature, knowledge and values. 

By studying the summary statements about the five philosophical schools, an individual can obtain an over- 
view of those schools with which he/she most strongly agrees or disagrees. An individual may be able to 
search out items or areas in the inventory which indicate that the person has agreed with two contradictory 
beliefs. 

Group Analysis . While MABE may provide insight for the individual who completes it, its primary purpose is 
to provide a general summary of groups of people. In attempting to determine the philosophic orientation of 
large groups of people, such as the total community of a school district, a random sample from that popula- 
tion would provide an accurate, unbiased way of assessing this group. The method of selecting this sample 
and the size of the sample needed are areas in which 'the administrator may need to obtain technical 
assistance. 

If a group analysis is to be made, each individual should be given a small tally sheet to write the five total 
scores from his/her scoring sheet. In compiling information from a group at a meeting, it is important to 
respect the needs of the individual for anonymity. Care should be taken to collect the tally sheets in such a 
way that no individual's scores are known. 

The administrator should add up the scores for each of the schools and divide by the number of persons who 
have taken the MABE inventory. This will give a group profile for responses within each philosophical school. 
The administrator can use the copy of the MABE Group Profile for the purpose of making a visual display of 
group results. For example, a transparency can be made. 

Technical Assistance . Users of MABE who wish to have technical assistance before they themselves ad- 
minister the survey to a group of persons in their community may obtain this by contacting the. Curriculum 
Director, Elementary and Secondary Education Section, Minnesota Department of Education, 550 Cedar 
Street, St Paul, Minnesota 55101. 

Those who would prefer to have an outsider come into their community to administer the inventory and to 
work with those who have taken it may also obtain this type of technical assistance from the Elementary and 
Secondary Education Section. 



Minnesota Analysis of Beliefs in Education 
Scoring Sheet 



IDEALISM 

CD 
10 

21 
35 



ESSENTIALISM 
4 

(9) 



PRAGMATISM 



EXISTENTIALISM 

® 
1 1 



BEHAVIORISM 
1 

19 
25 

37 



48 

64 
68 
71 

© 
84 
87 



51 



70 
72 
77 

© 
86 



Minnesota Analysis of Beliefs in Education 
Individual Profile 



IDEALISM 



ESSENTIALISM 



PRAGMATISM 



EXISTENTIALISM 



BEHAVIORISM 




Minnesota Analysis of Beliefs in Education 
Total Scores of an Individual Response 

Idealism Essentia I ism Pragmatism Existentialism Behaviorism 



Minnesota Analysis of Beliefs in Education 
Total Scores of an Individual Response 

Idealism Essentialism Pragmatism Existentialism Behaviorism 



Minnesota Analysis of Beliefs in Education 
Total Scores of an Individual Response 

Idealism Essentialism Pragmatism Existentialism Behaviorism 
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